Oct 31 12:31:01.728680 ip-10-0-135-41 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:31:02.174021 ip-10-0-135-41 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:02.174021 ip-10-0-135-41 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:31:02.174021 ip-10-0-135-41 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:02.174021 ip-10-0-135-41 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:02.180431 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180326 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.180431 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180429 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.180431 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180433 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.180431 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180437 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180441 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180444 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180446 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180449 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180452 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180456 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180462 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180465 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180468 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180471 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180474 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180477 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180479 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180482 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180485 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180497 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180500 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180503 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.180550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180505 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180508 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180511 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180513 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180516 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180518 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180521 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180524 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180527 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180529 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180532 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180535 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180538 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180540 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180543 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180546 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180550 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180553 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180555 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180558 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.180987 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180561 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180563 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180566 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180569 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180571 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180574 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180577 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180579 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180582 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180585 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180588 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180591 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180593 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180596 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180598 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180601 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180603 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180606 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180609 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.181501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180611 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180614 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180617 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180621 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180624 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180626 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180629 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180631 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180634 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180637 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180640 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180643 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180645 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180648 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180650 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180653 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180656 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180659 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180661 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180664 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.181949 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180666 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180669 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180674 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180678 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180682 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180684 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180687 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180689 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180692 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180695 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180697 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180700 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180703 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180705 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180708 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180711 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180714 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180716 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180719 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180721 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.182432 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180725 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180728 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180732 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180735 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.180738 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181531 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181538 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181541 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181545 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181548 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181551 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181554 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181556 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181559 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181562 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181565 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181568 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181571 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181574 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181576 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.182923 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181579 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181582 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181585 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181587 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181590 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181593 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181595 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181599 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181601 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181604 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181606 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181609 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181612 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181615 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181618 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181620 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181623 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181626 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181629 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.183486 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181631 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181634 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181637 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181639 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181642 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181644 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181647 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181649 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181652 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181656 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181658 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181661 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181663 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181666 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181668 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181671 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181673 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181676 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181678 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181681 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.184266 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181684 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181687 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181689 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181692 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181695 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181698 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181713 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181716 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181719 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181721 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181724 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181727 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181730 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181732 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181735 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181737 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181740 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181743 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181746 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.184985 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181748 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181752 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181755 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181757 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181760 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181763 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181765 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181768 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181770 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181773 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181776 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181778 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181783 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181786 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181788 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181791 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181794 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181797 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181800 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181803 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.185610 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181805 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181808 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181811 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181813 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181816 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181818 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181821 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181824 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181828 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181831 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181834 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.181839 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182341 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182347 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182351 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182354 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182357 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182361 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182366 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.186122 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182369 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182371 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182374 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182377 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182379 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182396 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182399 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182401 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182404 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182407 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182409 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182412 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182415 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182417 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182420 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182423 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182425 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182428 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182431 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182433 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.186597 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182436 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182439 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182442 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182445 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182447 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182450 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182452 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182455 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182458 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182460 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182463 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182467 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182470 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182473 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182475 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182479 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182482 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182486 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182489 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.187065 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182492 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182495 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182498 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182500 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182503 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182505 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182509 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182511 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182514 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182516 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182518 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182521 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182524 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182526 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182529 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182531 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182534 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182536 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182540 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.187570 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182544 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182547 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182550 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182553 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182556 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182559 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182562 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182564 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182567 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182570 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182572 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182575 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182577 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182580 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182582 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182585 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182587 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182590 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182593 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182596 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.188026 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182598 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182601 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182603 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182606 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182608 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182611 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182613 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182616 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182618 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182621 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182623 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182626 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182629 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182632 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182634 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182637 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182640 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182643 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182645 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182648 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.188550 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.182650 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182745 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182752 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182759 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182763 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182768 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182771 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182776 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182780 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182783 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182786 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182790 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182793 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182796 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182799 2580 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182803 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182806 2580 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182808 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182811 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182816 2580 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182818 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182821 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182825 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182828 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:31:02.189059 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182832 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182835 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182839 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182842 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182847 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182851 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182854 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182858 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182861 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182865 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182868 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182871 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182874 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182877 2580 flags.go:64] FLAG: --enable-server="true" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182880 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182885 2580 flags.go:64] FLAG: --event-burst="100" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182889 2580 flags.go:64] FLAG: --event-qps="50" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182891 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182895 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182898 2580 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182902 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182905 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182908 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182911 2580 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182914 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:31:02.189649 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182917 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182920 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182923 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182926 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182929 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182933 2580 flags.go:64] FLAG: --feature-gates="" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182937 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182940 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182943 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182946 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182949 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182954 2580 flags.go:64] FLAG: --help="false" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182959 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182963 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182966 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182969 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182973 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182976 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182979 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182982 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182985 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182988 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182991 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:31:02.190283 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182995 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.182998 2580 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183001 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183004 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183007 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183010 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183013 2580 flags.go:64] FLAG: --lock-file="" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183016 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183019 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183022 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183028 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183031 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183034 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183037 2580 flags.go:64] FLAG: --logging-format="text" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183040 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183044 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183047 2580 flags.go:64] FLAG: --manifest-url="" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183050 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183054 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183057 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183063 2580 flags.go:64] FLAG: --max-pods="110" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183068 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183072 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183075 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183079 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:31:02.190855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183082 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183085 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183088 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183096 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183099 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183102 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183105 2580 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183108 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183112 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183115 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183118 2580 flags.go:64] FLAG: --port="10250" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183121 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183124 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b7e1436149d8ecbb" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183127 2580 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183130 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183133 2580 flags.go:64] FLAG: --register-node="true" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183136 2580 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183140 2580 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183143 2580 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183146 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183149 2580 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183153 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183156 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183159 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183162 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183165 2580 flags.go:64] FLAG: --runonce="false" Oct 31 12:31:02.191453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183168 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183172 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183177 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183180 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183183 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183187 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183190 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183194 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183197 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183200 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183203 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183206 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183209 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183212 2580 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183214 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183220 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183223 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183226 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183230 2580 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183233 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183236 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183239 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183242 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183245 2580 flags.go:64] FLAG: --v="2" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183249 2580 flags.go:64] FLAG: --version="false" Oct 31 12:31:02.192113 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183254 2580 flags.go:64] FLAG: --vmodule="" Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183258 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.183261 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183859 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183881 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183887 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183894 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183903 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183908 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183912 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183917 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183922 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183926 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183931 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183935 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183940 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183944 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183949 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183953 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183962 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.192725 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183967 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183971 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183975 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183980 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183984 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183989 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183993 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.183998 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184002 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184007 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184012 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184016 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184025 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184030 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184034 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184039 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184045 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184051 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184055 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184060 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.193191 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184064 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184069 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184073 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184078 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184086 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184090 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184094 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184099 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184103 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184107 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184111 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184115 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184119 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184124 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184128 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184132 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184139 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184151 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184156 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.193680 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184163 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184169 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184174 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184179 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184184 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184188 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184192 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184196 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184201 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184206 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184218 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184222 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184226 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184230 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184234 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184239 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184243 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184247 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184252 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.194129 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184256 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184260 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184264 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184269 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184278 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184282 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184287 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184291 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184295 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184299 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184303 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184308 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184312 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184317 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184321 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184325 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184329 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184339 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184348 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.194581 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184352 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184357 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184361 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184366 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184371 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184377 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184381 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184404 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184409 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184414 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184423 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184427 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.184488 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184688 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184705 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184861 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.195012 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184866 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184870 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184874 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184877 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184880 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184882 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184885 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184888 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184891 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184894 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184896 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184900 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184902 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184905 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184908 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184911 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184913 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184916 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184919 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.195401 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184922 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184925 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184928 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184930 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184933 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184936 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184938 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184941 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184944 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184947 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184950 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184953 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184956 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184959 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184962 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184964 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184967 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184970 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184973 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184975 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.195848 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184978 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184981 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184984 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184986 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184989 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184992 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184995 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.184998 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185001 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185004 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185006 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185009 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185012 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185017 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185020 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185022 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185025 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185027 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185030 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185033 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.196314 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185036 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185038 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185041 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185046 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185049 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185052 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185055 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185058 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185060 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185063 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185065 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185068 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185072 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185077 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185080 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185085 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185089 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185092 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185095 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.196839 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185097 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185100 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185103 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185106 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185108 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185111 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185114 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185116 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185119 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185122 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185124 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185126 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185129 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185132 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185139 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185141 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185145 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185148 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185151 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.197283 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185155 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185158 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185160 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185163 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185166 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.185168 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.193053 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.193069 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.193085 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.193091 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193161 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193167 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193170 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193174 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193177 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193180 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193183 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193186 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193189 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193192 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.197786 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193195 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193198 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193201 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193204 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193207 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193210 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193212 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193215 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193217 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193220 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193222 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193227 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193231 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193234 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193238 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193242 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193245 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193249 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193252 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.198238 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193255 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193258 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193261 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193264 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193266 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193269 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193272 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193274 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193277 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193279 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193282 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193285 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193287 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193290 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193292 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193295 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193297 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193300 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193303 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.198705 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193305 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193308 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193310 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193313 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193315 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193318 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193321 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193324 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193327 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193329 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193332 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193335 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193338 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193341 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193344 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193346 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193349 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193351 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193354 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193357 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.199147 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193360 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193362 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193365 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193367 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193370 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193373 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193375 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193378 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193380 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193400 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193403 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193405 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193408 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193411 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193413 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193416 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193419 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193422 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193425 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.199629 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193427 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193430 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193433 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193435 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193438 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193440 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193444 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193447 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193450 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193453 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193455 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193458 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193461 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193463 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193466 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193469 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193471 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193474 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.200061 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193476 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.193505 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193647 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193652 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193655 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193658 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193661 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193663 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193666 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193669 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193672 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193675 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193677 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193681 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193683 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193686 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.200599 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193689 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193692 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193694 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193697 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193700 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193710 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193713 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193716 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193719 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193721 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193724 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193727 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193729 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193732 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193735 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193737 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193740 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193743 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193745 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193748 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.200968 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193751 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193753 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193756 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193758 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193763 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193766 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193769 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193772 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193775 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193778 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193781 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193783 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193786 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193788 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193792 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193795 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193797 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193800 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193805 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.201439 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193808 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193811 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193814 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193818 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193821 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193823 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193826 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193829 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193831 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193834 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193837 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193839 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193842 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193844 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193847 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193849 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193852 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193855 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193857 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.201876 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193860 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193862 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193865 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193868 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193871 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193873 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193876 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193879 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193881 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193883 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193886 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193888 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193891 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193894 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193897 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193900 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193902 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193905 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193907 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193910 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.202318 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193912 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193914 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193917 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193919 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193922 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193924 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193927 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193929 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193932 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193934 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193940 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193943 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193946 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.193948 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.193972 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194003 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:02.202788 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194008 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194011 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194014 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194018 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194021 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194024 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194026 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194029 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194031 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194034 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194037 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194040 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194043 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194045 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194048 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194050 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194053 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194055 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194058 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194061 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:02.203154 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194063 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194066 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194068 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194071 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194073 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194076 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194079 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194083 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194086 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194088 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194091 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194095 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194098 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194101 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194104 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194108 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194111 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194114 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194117 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:02.203639 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194119 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194122 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194125 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194127 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194130 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194133 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194137 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194141 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194144 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194147 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194150 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194153 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194156 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194158 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194161 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194163 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194166 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194170 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:02.204130 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194173 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194176 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194179 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194182 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194185 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194187 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194190 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194192 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194195 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194197 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194200 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194203 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194205 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194208 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194211 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194213 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194216 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194218 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194221 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194224 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:02.204587 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194226 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194229 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194231 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194234 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194237 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194239 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194242 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194244 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194247 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194250 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194252 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194256 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194258 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194261 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194265 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194268 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194270 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194273 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194276 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194278 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:02.205080 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194281 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194284 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194287 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194290 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194293 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194295 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194298 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:02.194300 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.194361 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.197626 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.198652 2580 server.go:1014] "Starting client certificate rotation" Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.198749 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.198788 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.200845 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:02.205593 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.203675 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:02.216502 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.216481 2580 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:31:02.221842 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.221816 2580 log.go:25] "Validated CRI v1 image API" Oct 31 12:31:02.223944 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.223916 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:31:02.226647 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.226625 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 db496fcc-61ff-49b6-8276-2d61b4f4b1e2:/dev/nvme0n1p4 f6b76be6-7865-4874-9bd0-c0ac52a690ae:/dev/nvme0n1p3] Oct 31 12:31:02.226713 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.226647 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 12:31:02.231820 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.231683 2580 manager.go:217] Machine: {Timestamp:2026-10-31 12:31:02.23028307 +0000 UTC m=+0.371968768 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104249 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e6546f1cd7437e5b80b8cbe7430e SystemUUID:ec28e654-6f1c-d743-7e5b-80b8cbe7430e BootID:d47cadb3-1bb7-47c8-817b-e1f7d5a2bad0 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:81:c1:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:81:c1:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:1f:cb:fa:86:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:31:02.231820 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.231818 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:31:02.231934 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.231896 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:31:02.235522 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.235499 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:31:02.235652 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.235521 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-41.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:31:02.235710 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.235654 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:31:02.235710 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.235663 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:31:02.235710 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.235674 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:02.235710 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.235683 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:02.236354 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.236329 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:31:02.236870 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.236859 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:31:02.241250 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241238 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:31:02.241370 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241360 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:31:02.241430 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241403 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:31:02.241430 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241413 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:31:02.241430 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241423 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:31:02.241535 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241432 2580 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:31:02.241535 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241440 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:31:02.241764 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.241752 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:02.244184 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.244167 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:31:02.251172 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.251158 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:02.252163 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252151 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:31:02.252207 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252187 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:31:02.252352 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252340 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:31:02.252412 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252355 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:31:02.252412 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252361 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:31:02.252412 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252367 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:31:02.252412 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252373 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:31:02.252412 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.252379 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:31:02.253166 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.253156 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:31:02.253197 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.253166 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:31:02.253197 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.253174 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:31:02.253197 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.253180 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:31:02.253197 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.253194 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:31:02.253315 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.253203 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:31:02.254856 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.254846 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:31:02.254856 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.254856 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:31:02.258540 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.258514 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-41.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 12:31:02.259008 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.258996 2580 server.go:1263] "Started kubelet" Oct 31 12:31:02.259139 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.259085 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:31:02.259234 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.259126 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:31:02.259234 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.259184 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:31:02.259922 ip-10-0-135-41 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:31:02.261019 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.261002 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:31:02.262672 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.261899 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:31:02.263878 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.263802 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-41.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 12:31:02.264678 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.264653 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 12:31:02.265999 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.265975 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 12:31:02.270719 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.270701 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:31:02.271814 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.270709 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-41.us-east-2.compute.internal.18e39d25ca6b29f8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:31:02.258960888 +0000 UTC m=+0.400646591,LastTimestamp:2026-10-31 12:31:02.258960888 +0000 UTC m=+0.400646591,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-41.us-east-2.compute.internal,}" Oct 31 12:31:02.272062 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.272049 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:31:02.272173 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.272153 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.272899 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.272881 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:31:02.272899 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.272899 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:31:02.273052 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.272884 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:31:02.273052 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273001 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:31:02.273052 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273009 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:31:02.273210 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273052 2580 factory.go:153] Registering CRI-O factory Oct 31 12:31:02.273210 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273090 2580 factory.go:223] Registration of the crio container factory successfully Oct 31 12:31:02.273210 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273136 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:31:02.273210 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273145 2580 factory.go:55] Registering systemd factory Oct 31 12:31:02.273210 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273153 2580 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:31:02.273210 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273168 2580 factory.go:103] Registering Raw factory Oct 31 12:31:02.273210 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.273182 2580 manager.go:1199] Started watching for new ooms in manager Oct 31 12:31:02.274137 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.274122 2580 manager.go:319] Starting recovery of all containers Oct 31 12:31:02.279474 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.277120 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-41.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 12:31:02.282677 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.282651 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 12:31:02.283721 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.283702 2580 manager.go:324] Recovery completed Oct 31 12:31:02.288199 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.288186 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:02.290119 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.290097 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:02.290189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.290137 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:02.290189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.290153 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:02.290737 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.290722 2580 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:31:02.290737 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.290735 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:31:02.290852 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.290723 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:02.290852 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.290754 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:31:02.291555 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.291540 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:02.291623 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.291569 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:02.291623 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.291581 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:02.292711 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.292642 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-41.us-east-2.compute.internal.18e39d25cc469691 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-41.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:31:02.290118289 +0000 UTC m=+0.431803993,LastTimestamp:2026-10-31 12:31:02.290118289 +0000 UTC m=+0.431803993,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-41.us-east-2.compute.internal,}" Oct 31 12:31:02.293121 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.293107 2580 policy_none.go:50] "Start" Oct 31 12:31:02.293173 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.293128 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:31:02.293173 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.293141 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:31:02.293670 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.293655 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5b8f8" Oct 31 12:31:02.295178 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.295166 2580 policy_none.go:44] "Start" Oct 31 12:31:02.305763 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.305663 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-41.us-east-2.compute.internal.18e39d25cc46f977 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-41.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:31:02.290143607 +0000 UTC m=+0.431829307,LastTimestamp:2026-10-31 12:31:02.290143607 +0000 UTC m=+0.431829307,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-41.us-east-2.compute.internal,}" Oct 31 12:31:02.307312 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.307292 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5b8f8" Oct 31 12:31:02.311143 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.311048 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-41.us-east-2.compute.internal.18e39d25cc47354e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-41.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:31:02.290158926 +0000 UTC m=+0.431844626,LastTimestamp:2026-10-31 12:31:02.290158926 +0000 UTC m=+0.431844626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-41.us-east-2.compute.internal,}" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.336007 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.338574 2580 manager.go:341] "Starting Device Plugin manager" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.339407 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.339622 2580 server.go:87] "Starting device plugin registration server" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.339901 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.339913 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.340027 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.340102 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.340109 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.340685 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.340723 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.343027 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.344146 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.344168 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.344186 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.344217 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 12:31:02.351520 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.347814 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:02.440323 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.440240 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:02.442256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.442240 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:02.442321 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.442276 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:02.442321 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.442286 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:02.442321 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.442312 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.444334 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.444316 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal"] Oct 31 12:31:02.444397 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.444372 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:02.445213 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.445199 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:02.445287 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.445223 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:02.445287 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.445232 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:02.447263 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.447251 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:02.447433 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.447415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.447478 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.447451 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:02.452699 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.452679 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.452699 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.452702 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-41.us-east-2.compute.internal\": node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.452982 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.452968 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:02.453050 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.452995 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:02.453050 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.452972 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:02.453050 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.453029 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:02.453050 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.453042 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:02.453050 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.453005 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:02.454995 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.454978 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.455082 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.455014 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:02.455786 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.455772 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:02.455860 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.455797 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:02.455860 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.455806 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:02.471294 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.471274 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" node="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.473098 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.473081 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" node="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.480900 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.480886 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.573774 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.573742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.573774 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.573775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84469bc02ae1e7d219895e3165c40bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"f84469bc02ae1e7d219895e3165c40bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.573967 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.573792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.581830 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.581807 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.674247 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.674161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.674247 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.674232 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.674445 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.674273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.674445 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.674295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84469bc02ae1e7d219895e3165c40bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"f84469bc02ae1e7d219895e3165c40bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.674445 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.674324 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.674445 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.674322 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84469bc02ae1e7d219895e3165c40bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"f84469bc02ae1e7d219895e3165c40bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.682279 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.682261 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.776009 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.775956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.778140 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:02.778119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:02.782972 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.782956 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.883230 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.883184 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:02.983766 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:02.983730 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:04.428591 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.428526 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:04.529110 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.529076 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:04.541674 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.541652 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:04.543677 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.543664 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 12:31:04.543783 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.543764 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:31:04.543816 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.543766 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 12:31:04.586012 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.585975 2580 apiserver.go:52] "Watching apiserver" Oct 31 12:31:04.607648 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.607601 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:04.608012 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.607997 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:31:04.608275 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.608258 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9tkzk","openshift-network-diagnostics/network-check-target-gjvxs","openshift-network-operator/iptables-alerter-6hdft","openshift-multus/multus-db7cd","kube-system/global-pull-secret-syncer-sk7rm","kube-system/konnectivity-agent-b4cc4","openshift-cluster-node-tuning-operator/tuned-7z7sx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6","openshift-dns/node-resolver-9v4bh","openshift-image-registry/node-ca-qkx64","openshift-multus/network-metrics-daemon-5kl4n","openshift-ovn-kubernetes/ovnkube-node-r8m9k"] Oct 31 12:31:04.611043 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.611018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.614333 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.614304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.615123 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.615099 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 12:31:04.615389 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.615369 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:31:04.615454 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.615368 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hk7cx\"" Oct 31 12:31:04.615454 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.615375 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:31:04.616482 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.616465 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:04.616565 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.616491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.617475 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.617458 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:04.617563 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.617465 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:04.617718 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.617702 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7phhl\"" Oct 31 12:31:04.618970 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.618956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.621437 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-host\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.621537 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a069e898-8264-4573-b885-2528dadb0305-hosts-file\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.621537 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.621537 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.621723 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-systemd\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.621723 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/f0f7cd28-a6f8-43af-b1dc-f85855564643-kube-api-access-sxhlm\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.621723 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.621723 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-registration-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.621911 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-conf\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.621911 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-lib-modules\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.621911 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pqtb\" (UniqueName: \"kubernetes.io/projected/a069e898-8264-4573-b885-2528dadb0305-kube-api-access-6pqtb\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.621911 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0554ee01-1433-4bd7-9a51-d74c757af46a-konnectivity-ca\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.621911 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-run\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622146 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-sys\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622146 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-socket-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.622146 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.621996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.622146 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxc57\" (UniqueName: \"kubernetes.io/projected/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kube-api-access-wxc57\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.622146 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-modprobe-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622146 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-kubernetes\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-var-lib-kubelet\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-sys-fs\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069e898-8264-4573-b885-2528dadb0305-tmp-dir\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysconfig\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-tuned\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-tmp\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-device-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.622360 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.622295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0554ee01-1433-4bd7-9a51-d74c757af46a-agent-certs\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.624042 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.624023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:04.624142 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.624121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:04.626778 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.626754 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.630588 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.629350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.632007 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.631979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.632106 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.632045 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:04.632448 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.632429 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:04.632511 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.632476 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 12:31:04.632547 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.632526 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:04.634627 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.634610 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.637254 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.637231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:04.637338 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.637306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:04.639928 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.639908 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal"] Oct 31 12:31:04.640013 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.639998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.640318 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.640301 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal"] Oct 31 12:31:04.640401 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.640387 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:04.657183 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.657139 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:26:03 +0000 UTC" deadline="2028-04-28 09:15:54.829686891 +0000 UTC" Oct 31 12:31:04.657183 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.657181 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13076h44m50.172508302s" Oct 31 12:31:04.660533 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.660514 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bqv79" Oct 31 12:31:04.672400 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.672377 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bqv79" Oct 31 12:31:04.715881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.715853 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:31:04.715881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.715860 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:31:04.716064 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.715946 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:31:04.716064 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.715973 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:04.716209 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716110 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:31:04.716209 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716108 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:31:04.716209 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716129 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:31:04.716209 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716140 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:31:04.716209 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716159 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:31:04.716209 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716176 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716220 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716261 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716270 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k457\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716327 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716339 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716344 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716351 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8c8p7\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716359 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716368 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716381 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:31:04.716462 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.716397 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bkb2g\"" Oct 31 12:31:04.717388 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717278 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:31:04.717388 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717343 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvpb\"" Oct 31 12:31:04.717388 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717354 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zfd9q\"" Oct 31 12:31:04.717388 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717369 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:31:04.717691 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717675 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:31:04.717761 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717720 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:31:04.717761 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717755 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:31:04.717861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717820 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmbl2\"" Oct 31 12:31:04.717912 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.717863 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w8vnc\"" Oct 31 12:31:04.718338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.718321 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:31:04.722545 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-kubernetes\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.722670 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-bin\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.722670 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-host\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.722670 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-cnibin\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.722781 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-systemd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.722781 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722664 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-kubernetes\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.722781 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.722781 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-config\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.722904 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-script-lib\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.722904 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-multus-certs\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.722904 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysconfig\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.722904 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-tuned\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723035 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722911 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysconfig\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723035 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-tmp\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723035 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-device-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.723035 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.722988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:04.723035 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-node-log\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.723196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-cni-binary-copy\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.723196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-env-overrides\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.723196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0554ee01-1433-4bd7-9a51-d74c757af46a-agent-certs\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.723196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723109 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-device-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.723196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723142 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:31:04.723196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.723414 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.723414 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723414 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-systemd\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723414 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723406 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-systemd\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723414 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723405 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-registration-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-etc-kubernetes\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-conf\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723520 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-registration-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723530 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-lib-modules\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-etc-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d409cf23-c109-4dfb-a569-5041531269dc-iptables-alerter-script\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-k8s-cni-cncf-io\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-bin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-conf-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0554ee01-1433-4bd7-9a51-d74c757af46a-konnectivity-ca\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.723733 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723728 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-conf\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-sys\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-socket-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxc57\" (UniqueName: \"kubernetes.io/projected/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kube-api-access-wxc57\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723871 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-lib-modules\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-socket-dir-parent\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-netns\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-var-lib-kubelet\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-systemd-units\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-netd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723956 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-multus-daemon-config\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.723991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-sys\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pc8z\" (UniqueName: \"kubernetes.io/projected/9c7c8096-81d8-4c14-b102-08569d25da31-kube-api-access-2pc8z\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-kubelet-config\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.724324 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-system-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-sys-fs\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069e898-8264-4573-b885-2528dadb0305-tmp-dir\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724079 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-socket-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-slash\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovn-node-metrics-cert\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724185 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-sys-fs\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724203 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0554ee01-1433-4bd7-9a51-d74c757af46a-konnectivity-ca\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724315 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-var-lib-kubelet\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724402 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-kubelet\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724403 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069e898-8264-4573-b885-2528dadb0305-tmp-dir\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-system-cni-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757tm\" (UniqueName: \"kubernetes.io/projected/59030007-53c1-42c9-baec-a4d2ec90f3dd-kube-api-access-757tm\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-kubelet\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99f5v\" (UniqueName: \"kubernetes.io/projected/d409cf23-c109-4dfb-a569-5041531269dc-kube-api-access-99f5v\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.725114 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-multus\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-dbus\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-host\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a069e898-8264-4573-b885-2528dadb0305-hosts-file\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724796 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-host\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-ovn\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-log-socket\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724881 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a069e898-8264-4573-b885-2528dadb0305-hosts-file\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409cf23-c109-4dfb-a569-5041531269dc-host-slash\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/f0f7cd28-a6f8-43af-b1dc-f85855564643-kube-api-access-sxhlm\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.724990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-var-lib-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv66w\" (UniqueName: \"kubernetes.io/projected/4a8a0aad-3971-455a-8fda-f40c28d8bc13-kube-api-access-fv66w\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-hostroot\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7tbp\" (UniqueName: \"kubernetes.io/projected/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-kube-api-access-n7tbp\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqtb\" (UniqueName: \"kubernetes.io/projected/a069e898-8264-4573-b885-2528dadb0305-kube-api-access-6pqtb\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-serviceca\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.725909 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-run\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhdx\" (UniqueName: \"kubernetes.io/projected/fb0b76f8-eb8d-45ee-afec-b393223b0504-kube-api-access-wzhdx\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725468 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-run\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-netns\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-cnibin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-modprobe-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.725845 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-modprobe-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.726679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.726627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-tmp\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.726925 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.726735 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-tuned\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.727041 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.727023 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0554ee01-1433-4bd7-9a51-d74c757af46a-agent-certs\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.736536 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.736511 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/f0f7cd28-a6f8-43af-b1dc-f85855564643-kube-api-access-sxhlm\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.736644 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.736540 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqtb\" (UniqueName: \"kubernetes.io/projected/a069e898-8264-4573-b885-2528dadb0305-kube-api-access-6pqtb\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:04.737400 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.737380 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxc57\" (UniqueName: \"kubernetes.io/projected/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kube-api-access-wxc57\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.806501 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:04.806458 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7397e52f9425481ec78d95c85d33a100.slice/crio-b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4 WatchSource:0}: Error finding container b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4: Status 404 returned error can't find the container with id b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4 Oct 31 12:31:04.806921 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:04.806901 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84469bc02ae1e7d219895e3165c40bd.slice/crio-842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02 WatchSource:0}: Error finding container 842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02: Status 404 returned error can't find the container with id 842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02 Oct 31 12:31:04.812280 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.812263 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:31:04.826017 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.825985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-kubelet\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826137 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99f5v\" (UniqueName: \"kubernetes.io/projected/d409cf23-c109-4dfb-a569-5041531269dc-kube-api-access-99f5v\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.826137 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-multus\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.826137 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-dbus\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.826137 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-ovn\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826137 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-log-socket\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826137 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409cf23-c109-4dfb-a569-5041531269dc-host-slash\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-var-lib-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv66w\" (UniqueName: \"kubernetes.io/projected/4a8a0aad-3971-455a-8fda-f40c28d8bc13-kube-api-access-fv66w\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826192 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-multus\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-ovn\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-dbus\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-hostroot\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826242 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409cf23-c109-4dfb-a569-5041531269dc-host-slash\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826271 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-hostroot\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826278 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-var-lib-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826195 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-log-socket\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826307 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-kubelet\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7tbp\" (UniqueName: \"kubernetes.io/projected/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-kube-api-access-n7tbp\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-serviceca\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhdx\" (UniqueName: \"kubernetes.io/projected/fb0b76f8-eb8d-45ee-afec-b393223b0504-kube-api-access-wzhdx\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-netns\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.826483 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826477 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826520 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-netns\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-cnibin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-bin\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-host\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-cnibin\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-systemd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826727 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-host\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826727 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-cnibin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826772 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826799 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-bin\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-config\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-script-lib\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-cnibin\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-multus-certs\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.827202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826887 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826921 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-multus-certs\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826925 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-node-log\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-cni-binary-copy\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.826973 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-node-log\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.826999 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827031 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-systemd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.827076 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs podName:fb0b76f8-eb8d-45ee-afec-b393223b0504 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:05.327043871 +0000 UTC m=+2.124481931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs") pod "network-metrics-daemon-5kl4n" (UID: "fb0b76f8-eb8d-45ee-afec-b393223b0504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-env-overrides\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-etc-kubernetes\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-etc-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d409cf23-c109-4dfb-a569-5041531269dc-iptables-alerter-script\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-k8s-cni-cncf-io\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828058 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-bin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-conf-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-socket-dir-parent\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-netns\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-systemd-units\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-netd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-multus-daemon-config\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc8z\" (UniqueName: \"kubernetes.io/projected/9c7c8096-81d8-4c14-b102-08569d25da31-kube-api-access-2pc8z\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-kubelet-config\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-system-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-slash\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovn-node-metrics-cert\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-serviceca\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-kubelet\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-system-cni-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.828562 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827710 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827751 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-netns\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827774 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-systemd-units\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757tm\" (UniqueName: \"kubernetes.io/projected/59030007-53c1-42c9-baec-a4d2ec90f3dd-kube-api-access-757tm\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-netd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827890 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-kubelet-config\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827941 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-system-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.827989 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-slash\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.828021 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-conf-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.828845 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-env-overrides\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.828901 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-config\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.828917 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-script-lib\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.828926 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.828941 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.828978 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret podName:8cea9a4e-e914-40b6-9237-345be4aff28c nodeName:}" failed. No retries permitted until 2026-10-31 12:31:05.328961375 +0000 UTC m=+2.126399436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret") pod "global-pull-secret-syncer-sk7rm" (UID: "8cea9a4e-e914-40b6-9237-345be4aff28c") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.828905 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-cni-binary-copy\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829338 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829070 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829071 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829120 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-etc-kubernetes\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829157 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-socket-dir-parent\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829166 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-k8s-cni-cncf-io\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829189 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-bin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829199 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-system-cni-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-kubelet\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829160 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-etc-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829436 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d409cf23-c109-4dfb-a569-5041531269dc-iptables-alerter-script\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829467 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.829881 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.829492 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-multus-daemon-config\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.830937 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.830921 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovn-node-metrics-cert\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.841660 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.841614 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:04.841660 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.841658 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:04.841660 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.841668 2580 projected.go:196] Error preparing data for projected volume kube-api-access-c4sk5 for pod openshift-network-diagnostics/network-check-target-gjvxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:04.841874 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:04.841734 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5 podName:f05d824e-3be1-46f6-8df0-2ade9675e534 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:05.341707179 +0000 UTC m=+2.139145236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c4sk5" (UniqueName: "kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5") pod "network-check-target-gjvxs" (UID: "f05d824e-3be1-46f6-8df0-2ade9675e534") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:04.842872 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.842840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99f5v\" (UniqueName: \"kubernetes.io/projected/d409cf23-c109-4dfb-a569-5041531269dc-kube-api-access-99f5v\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:04.844264 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.844237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv66w\" (UniqueName: \"kubernetes.io/projected/4a8a0aad-3971-455a-8fda-f40c28d8bc13-kube-api-access-fv66w\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:04.844415 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.844396 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc8z\" (UniqueName: \"kubernetes.io/projected/9c7c8096-81d8-4c14-b102-08569d25da31-kube-api-access-2pc8z\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:04.844493 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.844470 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-757tm\" (UniqueName: \"kubernetes.io/projected/59030007-53c1-42c9-baec-a4d2ec90f3dd-kube-api-access-757tm\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:04.844493 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.844487 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhdx\" (UniqueName: \"kubernetes.io/projected/fb0b76f8-eb8d-45ee-afec-b393223b0504-kube-api-access-wzhdx\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:04.844646 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.844615 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7tbp\" (UniqueName: \"kubernetes.io/projected/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-kube-api-access-n7tbp\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:04.947813 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.947734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:04.957394 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:04.957371 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0554ee01_1433_4bd7_9a51_d74c757af46a.slice/crio-74722bde183b22d5645720fd42e05bd85a6b4d8e89e24e382af767b234e855aa WatchSource:0}: Error finding container 74722bde183b22d5645720fd42e05bd85a6b4d8e89e24e382af767b234e855aa: Status 404 returned error can't find the container with id 74722bde183b22d5645720fd42e05bd85a6b4d8e89e24e382af767b234e855aa Oct 31 12:31:04.966744 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.966725 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:04.973719 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.973700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:04.974763 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:04.974741 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f7cd28_a6f8_43af_b1dc_f85855564643.slice/crio-4a180d2bee65b08d04827e8f79f8e6d775f34785a58b17688772447e54d8516d WatchSource:0}: Error finding container 4a180d2bee65b08d04827e8f79f8e6d775f34785a58b17688772447e54d8516d: Status 404 returned error can't find the container with id 4a180d2bee65b08d04827e8f79f8e6d775f34785a58b17688772447e54d8516d Oct 31 12:31:04.982767 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:04.982747 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f23f959_82b3_48cc_8f17_917d8c8c6f6b.slice/crio-a93a0b3166b3bfc67fcb0fe80b254a7438f49ff4f300cd62bd088f3168dffb64 WatchSource:0}: Error finding container a93a0b3166b3bfc67fcb0fe80b254a7438f49ff4f300cd62bd088f3168dffb64: Status 404 returned error can't find the container with id a93a0b3166b3bfc67fcb0fe80b254a7438f49ff4f300cd62bd088f3168dffb64 Oct 31 12:31:04.991640 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:04.991613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:05.000011 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:04.999984 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda069e898_8264_4573_b885_2528dadb0305.slice/crio-fce15112b54ab63d32596f08ee801da328b9b48d5522eaa3f1f7e7d3489ea9fc WatchSource:0}: Error finding container fce15112b54ab63d32596f08ee801da328b9b48d5522eaa3f1f7e7d3489ea9fc: Status 404 returned error can't find the container with id fce15112b54ab63d32596f08ee801da328b9b48d5522eaa3f1f7e7d3489ea9fc Oct 31 12:31:05.026573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.026552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:05.034142 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:05.034123 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59030007_53c1_42c9_baec_a4d2ec90f3dd.slice/crio-1375b84931ed41b9380d19cb4f2dfad25207356b545b746f32cfb042a9706ac3 WatchSource:0}: Error finding container 1375b84931ed41b9380d19cb4f2dfad25207356b545b746f32cfb042a9706ac3: Status 404 returned error can't find the container with id 1375b84931ed41b9380d19cb4f2dfad25207356b545b746f32cfb042a9706ac3 Oct 31 12:31:05.035468 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.035453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:05.043681 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:05.043659 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd409cf23_c109_4dfb_a569_5041531269dc.slice/crio-f33883c96c28647af1b557d2939ca7f9aaa5215fd7bb2d2875a7a45dacbe96ba WatchSource:0}: Error finding container f33883c96c28647af1b557d2939ca7f9aaa5215fd7bb2d2875a7a45dacbe96ba: Status 404 returned error can't find the container with id f33883c96c28647af1b557d2939ca7f9aaa5215fd7bb2d2875a7a45dacbe96ba Oct 31 12:31:05.048870 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.048852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-db7cd" Oct 31 12:31:05.057242 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:05.057224 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7c8096_81d8_4c14_b102_08569d25da31.slice/crio-4853821b0334d14c66f20cc1ff71c57dbd939417156d20a6f09c401610e96076 WatchSource:0}: Error finding container 4853821b0334d14c66f20cc1ff71c57dbd939417156d20a6f09c401610e96076: Status 404 returned error can't find the container with id 4853821b0334d14c66f20cc1ff71c57dbd939417156d20a6f09c401610e96076 Oct 31 12:31:05.065233 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.065218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:05.072081 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.072065 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:05.073684 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:05.073665 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcc043c_d8ac_4f2a_ae8d_3e233db2842d.slice/crio-915733798ecb455c1a10ae60de736827ff18469f26da3a6cbe871bc903d873b1 WatchSource:0}: Error finding container 915733798ecb455c1a10ae60de736827ff18469f26da3a6cbe871bc903d873b1: Status 404 returned error can't find the container with id 915733798ecb455c1a10ae60de736827ff18469f26da3a6cbe871bc903d873b1 Oct 31 12:31:05.086536 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.086519 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:05.095538 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:05.095520 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8a0aad_3971_455a_8fda_f40c28d8bc13.slice/crio-353ac1b3869e6ffb9011f953aaf415aadf741369d91fdd9b0176bece61c6c29a WatchSource:0}: Error finding container 353ac1b3869e6ffb9011f953aaf415aadf741369d91fdd9b0176bece61c6c29a: Status 404 returned error can't find the container with id 353ac1b3869e6ffb9011f953aaf415aadf741369d91fdd9b0176bece61c6c29a Oct 31 12:31:05.332149 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.332119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:05.332363 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.332205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:05.332363 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.332273 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:05.332363 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.332279 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:05.332363 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.332334 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs podName:fb0b76f8-eb8d-45ee-afec-b393223b0504 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:06.332321334 +0000 UTC m=+3.129759379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs") pod "network-metrics-daemon-5kl4n" (UID: "fb0b76f8-eb8d-45ee-afec-b393223b0504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:05.332363 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.332352 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret podName:8cea9a4e-e914-40b6-9237-345be4aff28c nodeName:}" failed. No retries permitted until 2026-10-31 12:31:06.332345881 +0000 UTC m=+3.129783924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret") pod "global-pull-secret-syncer-sk7rm" (UID: "8cea9a4e-e914-40b6-9237-345be4aff28c") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:05.430768 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.430731 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:05.433170 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.433140 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:05.433304 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.433243 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:05.433304 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.433264 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:05.433304 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.433276 2580 projected.go:196] Error preparing data for projected volume kube-api-access-c4sk5 for pod openshift-network-diagnostics/network-check-target-gjvxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:05.433465 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:05.433343 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5 podName:f05d824e-3be1-46f6-8df0-2ade9675e534 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:06.43332517 +0000 UTC m=+3.230763215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4sk5" (UniqueName: "kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5") pod "network-check-target-gjvxs" (UID: "f05d824e-3be1-46f6-8df0-2ade9675e534") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:05.675197 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.675107 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:04 +0000 UTC" deadline="2028-07-25 20:59:53.860850898 +0000 UTC" Oct 31 12:31:05.675197 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.675152 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15200h28m48.185701748s" Oct 31 12:31:05.707187 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.707128 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hdft" event={"ID":"d409cf23-c109-4dfb-a569-5041531269dc","Type":"ContainerStarted","Data":"f33883c96c28647af1b557d2939ca7f9aaa5215fd7bb2d2875a7a45dacbe96ba"} Oct 31 12:31:05.711266 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.711232 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b4cc4" event={"ID":"0554ee01-1433-4bd7-9a51-d74c757af46a","Type":"ContainerStarted","Data":"74722bde183b22d5645720fd42e05bd85a6b4d8e89e24e382af767b234e855aa"} Oct 31 12:31:05.715906 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.715884 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"f84469bc02ae1e7d219895e3165c40bd","Type":"ContainerStarted","Data":"842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02"} Oct 31 12:31:05.727401 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.727376 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-db7cd" event={"ID":"9c7c8096-81d8-4c14-b102-08569d25da31","Type":"ContainerStarted","Data":"4853821b0334d14c66f20cc1ff71c57dbd939417156d20a6f09c401610e96076"} Oct 31 12:31:05.732041 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.731959 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerStarted","Data":"1375b84931ed41b9380d19cb4f2dfad25207356b545b746f32cfb042a9706ac3"} Oct 31 12:31:05.737586 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.737520 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9v4bh" event={"ID":"a069e898-8264-4573-b885-2528dadb0305","Type":"ContainerStarted","Data":"fce15112b54ab63d32596f08ee801da328b9b48d5522eaa3f1f7e7d3489ea9fc"} Oct 31 12:31:05.742288 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.742219 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" event={"ID":"2f23f959-82b3-48cc-8f17-917d8c8c6f6b","Type":"ContainerStarted","Data":"a93a0b3166b3bfc67fcb0fe80b254a7438f49ff4f300cd62bd088f3168dffb64"} Oct 31 12:31:05.746600 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.746539 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" event={"ID":"f0f7cd28-a6f8-43af-b1dc-f85855564643","Type":"ContainerStarted","Data":"4a180d2bee65b08d04827e8f79f8e6d775f34785a58b17688772447e54d8516d"} Oct 31 12:31:05.752908 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.752845 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"7397e52f9425481ec78d95c85d33a100","Type":"ContainerStarted","Data":"b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4"} Oct 31 12:31:05.759166 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.759142 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"353ac1b3869e6ffb9011f953aaf415aadf741369d91fdd9b0176bece61c6c29a"} Oct 31 12:31:05.765532 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:05.765507 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qkx64" event={"ID":"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d","Type":"ContainerStarted","Data":"915733798ecb455c1a10ae60de736827ff18469f26da3a6cbe871bc903d873b1"} Oct 31 12:31:06.234452 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.234411 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:06.341394 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.341354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:06.341559 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.341417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:06.341652 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.341621 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:06.341714 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.341693 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret podName:8cea9a4e-e914-40b6-9237-345be4aff28c nodeName:}" failed. No retries permitted until 2026-10-31 12:31:08.341676047 +0000 UTC m=+5.139114092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret") pod "global-pull-secret-syncer-sk7rm" (UID: "8cea9a4e-e914-40b6-9237-345be4aff28c") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:06.342105 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.342087 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:06.342197 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.342148 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs podName:fb0b76f8-eb8d-45ee-afec-b393223b0504 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:08.342124676 +0000 UTC m=+5.139562723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs") pod "network-metrics-daemon-5kl4n" (UID: "fb0b76f8-eb8d-45ee-afec-b393223b0504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:06.443026 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.442990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:06.443217 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.443197 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:06.443286 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.443226 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:06.443286 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.443240 2580 projected.go:196] Error preparing data for projected volume kube-api-access-c4sk5 for pod openshift-network-diagnostics/network-check-target-gjvxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:06.443394 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.443298 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5 podName:f05d824e-3be1-46f6-8df0-2ade9675e534 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:08.443278687 +0000 UTC m=+5.240716750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4sk5" (UniqueName: "kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5") pod "network-check-target-gjvxs" (UID: "f05d824e-3be1-46f6-8df0-2ade9675e534") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:06.675871 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.675737 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:04 +0000 UTC" deadline="2028-05-02 13:38:22.337093677 +0000 UTC" Oct 31 12:31:06.675871 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.675797 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13177h7m15.661301463s" Oct 31 12:31:06.689829 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.688964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:06.689829 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.689134 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:06.689829 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.689559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:06.689829 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.689671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:06.689829 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:06.689732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:06.689829 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:06.689795 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:08.360692 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:08.359868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:08.360692 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:08.359918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:08.360692 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.360075 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:08.360692 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.360137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret podName:8cea9a4e-e914-40b6-9237-345be4aff28c nodeName:}" failed. No retries permitted until 2026-10-31 12:31:12.360117284 +0000 UTC m=+9.157555331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret") pod "global-pull-secret-syncer-sk7rm" (UID: "8cea9a4e-e914-40b6-9237-345be4aff28c") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:08.360692 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.360575 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:08.360692 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.360628 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs podName:fb0b76f8-eb8d-45ee-afec-b393223b0504 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:12.360610437 +0000 UTC m=+9.158048500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs") pod "network-metrics-daemon-5kl4n" (UID: "fb0b76f8-eb8d-45ee-afec-b393223b0504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:08.461517 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:08.460931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:08.461517 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.461112 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:08.461517 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.461136 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:08.461517 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.461149 2580 projected.go:196] Error preparing data for projected volume kube-api-access-c4sk5 for pod openshift-network-diagnostics/network-check-target-gjvxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:08.461517 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.461201 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5 podName:f05d824e-3be1-46f6-8df0-2ade9675e534 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:12.461187033 +0000 UTC m=+9.258625091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4sk5" (UniqueName: "kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5") pod "network-check-target-gjvxs" (UID: "f05d824e-3be1-46f6-8df0-2ade9675e534") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:08.689700 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:08.689029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:08.689700 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.689188 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:08.689700 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:08.689585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:08.689973 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.689873 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:08.689973 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:08.689923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:08.690075 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:08.690005 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:09.808715 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.808413 2580 generic.go:356] "Generic (PLEG): container finished" podID="7397e52f9425481ec78d95c85d33a100" containerID="569d1999e691fe21bf82eaddbfdd3963d6ecb02badf5a7ef84f352ab5f2205f0" exitCode=0 Oct 31 12:31:09.809174 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.808736 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"7397e52f9425481ec78d95c85d33a100","Type":"ContainerDied","Data":"569d1999e691fe21bf82eaddbfdd3963d6ecb02badf5a7ef84f352ab5f2205f0"} Oct 31 12:31:09.815084 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.815050 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b4cc4" event={"ID":"0554ee01-1433-4bd7-9a51-d74c757af46a","Type":"ContainerStarted","Data":"6201343037fafc447c048af550d1b98607cf895e68b7f616ffc27c560e7c3245"} Oct 31 12:31:09.819623 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.819592 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"f84469bc02ae1e7d219895e3165c40bd","Type":"ContainerStarted","Data":"c6e42bbcea2a8e54271a477416a7b0b8a29dd7f9f2057cda4a8fe3a5edbcebe3"} Oct 31 12:31:09.823694 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.823665 2580 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="60eae7931cf1ad525d2bfb83395db0bbc8f706d1675774329f2a3e159cb41d60" exitCode=0 Oct 31 12:31:09.823791 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.823709 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerDied","Data":"60eae7931cf1ad525d2bfb83395db0bbc8f706d1675774329f2a3e159cb41d60"} Oct 31 12:31:09.863675 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.862457 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b4cc4" podStartSLOduration=2.419742173 podStartE2EDuration="6.862435658s" totalImagesPullingTime="4.442693485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:09.846141069 +0000 UTC m=+6.643579134" watchObservedRunningTime="2026-10-31 12:31:09.862435658 +0000 UTC m=+6.659873727" Oct 31 12:31:09.869133 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:09.868264 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" podStartSLOduration=5.868213593 podStartE2EDuration="5.868213593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:09.864340335 +0000 UTC m=+6.661778396" watchObservedRunningTime="2026-10-31 12:31:09.868213593 +0000 UTC m=+6.665651650" Oct 31 12:31:10.690112 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:10.689242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:10.690112 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:10.689382 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:10.690112 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:10.689824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:10.690112 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:10.689936 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:10.690112 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:10.690008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:10.690112 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:10.690080 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:10.829413 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:10.828601 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qkx64" event={"ID":"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d","Type":"ContainerStarted","Data":"e5d12b3852f44cc7aafc02605164f3f6b0253b4545853901a8737cbaf130156d"} Oct 31 12:31:10.834252 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:10.833302 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"7397e52f9425481ec78d95c85d33a100","Type":"ContainerStarted","Data":"763139fd980070db425f6480f037647a6d030d2679ef172db7d0761d2fc361b8"} Oct 31 12:31:10.869071 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:10.868399 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" podStartSLOduration=6.868382863 podStartE2EDuration="6.868382863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:10.867733435 +0000 UTC m=+7.665171502" watchObservedRunningTime="2026-10-31 12:31:10.868382863 +0000 UTC m=+7.665820931" Oct 31 12:31:10.869071 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:10.868643 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qkx64" podStartSLOduration=1.422703447 podStartE2EDuration="6.868622912s" totalImagesPullingTime="5.445919465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:10.849078369 +0000 UTC m=+7.646516436" watchObservedRunningTime="2026-10-31 12:31:10.868622912 +0000 UTC m=+7.666060978" Oct 31 12:31:11.838915 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:11.838877 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" event={"ID":"2f23f959-82b3-48cc-8f17-917d8c8c6f6b","Type":"ContainerStarted","Data":"c164490a497d1a0d5e4a4487db3bc4c01a25c86b7e83aa293b8cb706987e06e6"} Oct 31 12:31:12.396738 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.396705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:12.396907 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.396768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:12.396907 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.396857 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:12.397017 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.396916 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs podName:fb0b76f8-eb8d-45ee-afec-b393223b0504 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:20.396898869 +0000 UTC m=+17.194336918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs") pod "network-metrics-daemon-5kl4n" (UID: "fb0b76f8-eb8d-45ee-afec-b393223b0504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:12.397338 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.397315 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:12.397437 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.397378 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret podName:8cea9a4e-e914-40b6-9237-345be4aff28c nodeName:}" failed. No retries permitted until 2026-10-31 12:31:20.39736278 +0000 UTC m=+17.194800829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret") pod "global-pull-secret-syncer-sk7rm" (UID: "8cea9a4e-e914-40b6-9237-345be4aff28c") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:12.499605 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.498940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:12.499605 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.499113 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:12.499605 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.499131 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:12.499605 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.499143 2580 projected.go:196] Error preparing data for projected volume kube-api-access-c4sk5 for pod openshift-network-diagnostics/network-check-target-gjvxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:12.499605 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.499203 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5 podName:f05d824e-3be1-46f6-8df0-2ade9675e534 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:20.499185041 +0000 UTC m=+17.296623098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4sk5" (UniqueName: "kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5") pod "network-check-target-gjvxs" (UID: "f05d824e-3be1-46f6-8df0-2ade9675e534") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:12.689704 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.689665 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:12.689932 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.689714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:12.689932 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.689816 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:12.689932 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.689875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:12.690143 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.690001 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:12.690143 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:12.690084 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:12.788903 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.784727 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:12.788903 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.788354 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:12.845745 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.845708 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-db7cd" event={"ID":"9c7c8096-81d8-4c14-b102-08569d25da31","Type":"ContainerStarted","Data":"be014adfd7cf851ccfa46ab996536eb756e23ae53b2d69bc9602976a066c4136"} Oct 31 12:31:12.867327 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:12.867268 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-db7cd" podStartSLOduration=1.223288602 podStartE2EDuration="8.867247224s" totalImagesPullingTime="7.643958622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:12.866933571 +0000 UTC m=+9.664371638" watchObservedRunningTime="2026-10-31 12:31:12.867247224 +0000 UTC m=+9.664685339" Oct 31 12:31:13.313466 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:13.313440 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:31:13.687795 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:13.687441 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:31:13.313459642Z","UUID":"2c971b00-af32-4340-8c2e-174c64df9713","Handler":null,"Name":"","Endpoint":""} Oct 31 12:31:13.689441 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:13.689409 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:31:13.689441 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:13.689444 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:31:13.849950 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:13.849901 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9v4bh" event={"ID":"a069e898-8264-4573-b885-2528dadb0305","Type":"ContainerStarted","Data":"b909729625d76f8e023781b05a35a665d926f36ee84f1ba3aa266095afdae0ad"} Oct 31 12:31:13.852855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:13.852824 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" event={"ID":"2f23f959-82b3-48cc-8f17-917d8c8c6f6b","Type":"ContainerStarted","Data":"858c4b71e178c92e52ad7edab79e82599673f9e7f28dffd3c415a78d2811706d"} Oct 31 12:31:13.866212 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:13.866164 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9v4bh" podStartSLOduration=2.826650113 podStartE2EDuration="10.866150739s" totalImagesPullingTime="8.039500626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:13.865946401 +0000 UTC m=+10.663384468" watchObservedRunningTime="2026-10-31 12:31:13.866150739 +0000 UTC m=+10.663588806" Oct 31 12:31:14.689213 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:14.689177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:14.689362 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:14.689324 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:14.689648 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:14.689615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:14.689760 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:14.689702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:14.689760 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:14.689742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:14.689836 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:14.689783 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:14.856365 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:14.856330 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" event={"ID":"2f23f959-82b3-48cc-8f17-917d8c8c6f6b","Type":"ContainerStarted","Data":"e71d1e59e6c3bd6e58775e4c0db23cb121b5c017a84774d55f656c79f2d2ceed"} Oct 31 12:31:14.858292 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:14.858258 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hdft" event={"ID":"d409cf23-c109-4dfb-a569-5041531269dc","Type":"ContainerStarted","Data":"4bd212d9c1d1b8fb00381c0660aabfba2b60fe7983a7a1eba07a28205be0a3af"} Oct 31 12:31:14.877415 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:14.877359 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" podStartSLOduration=2.52457233 podStartE2EDuration="11.87734287s" totalImagesPullingTime="9.35277054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:14.875649746 +0000 UTC m=+11.673087813" watchObservedRunningTime="2026-10-31 12:31:14.87734287 +0000 UTC m=+11.674780936" Oct 31 12:31:14.891460 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:14.891412 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6hdft" podStartSLOduration=2.7302254980000003 podStartE2EDuration="10.891395077s" totalImagesPullingTime="8.161169579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:14.891270107 +0000 UTC m=+11.688708173" watchObservedRunningTime="2026-10-31 12:31:14.891395077 +0000 UTC m=+11.688833145" Oct 31 12:31:15.862510 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:15.862425 2580 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="0831b148fcfdf420ad52902629556a0e2b459bfd1f63e19380d789db03b08601" exitCode=0 Oct 31 12:31:15.863344 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:15.862514 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerDied","Data":"0831b148fcfdf420ad52902629556a0e2b459bfd1f63e19380d789db03b08601"} Oct 31 12:31:15.864143 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:15.864117 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" event={"ID":"f0f7cd28-a6f8-43af-b1dc-f85855564643","Type":"ContainerStarted","Data":"189767de64c689483697d99e873fc15c80bf29a2331187561d36b5e6286ea690"} Oct 31 12:31:15.907987 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:15.907944 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" podStartSLOduration=2.926701394 podStartE2EDuration="12.90793175s" totalImagesPullingTime="9.981230356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:15.907673359 +0000 UTC m=+12.705111426" watchObservedRunningTime="2026-10-31 12:31:15.90793175 +0000 UTC m=+12.705369815" Oct 31 12:31:16.688809 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:16.688772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:16.688809 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:16.688803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:16.688990 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:16.688789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:16.688990 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:16.688910 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:16.689046 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:16.689015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:16.689086 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:16.689068 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:17.441915 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:17.441834 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:17.442395 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:17.442033 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:31:17.442725 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:17.442702 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:17.869850 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:17.869647 2580 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="fed21df9ed8b0a30361494cfc19c5e279ff1b6f55e24e81ac5e00b045a1f499f" exitCode=0 Oct 31 12:31:17.870041 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:17.869664 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerDied","Data":"fed21df9ed8b0a30361494cfc19c5e279ff1b6f55e24e81ac5e00b045a1f499f"} Oct 31 12:31:18.688700 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:18.688659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:18.689196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:18.688712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:18.689196 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:18.688798 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:18.689196 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:18.688664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:18.689196 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:18.688905 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:18.689196 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:18.688962 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:19.876922 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:19.876885 2580 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="f1c581db68b380f5e00106fa2ddeaf81dc7d6691e3dca743a85b274db64a795b" exitCode=0 Oct 31 12:31:19.877437 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:19.876950 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerDied","Data":"f1c581db68b380f5e00106fa2ddeaf81dc7d6691e3dca743a85b274db64a795b"} Oct 31 12:31:20.461569 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.461532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:20.461747 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.461578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:20.461747 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.461705 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:20.461814 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.461768 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs podName:fb0b76f8-eb8d-45ee-afec-b393223b0504 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:36.46175267 +0000 UTC m=+33.259190714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs") pod "network-metrics-daemon-5kl4n" (UID: "fb0b76f8-eb8d-45ee-afec-b393223b0504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 12:31:20.461814 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.461705 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:20.461874 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.461834 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret podName:8cea9a4e-e914-40b6-9237-345be4aff28c nodeName:}" failed. No retries permitted until 2026-10-31 12:31:36.461823406 +0000 UTC m=+33.259261464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret") pod "global-pull-secret-syncer-sk7rm" (UID: "8cea9a4e-e914-40b6-9237-345be4aff28c") : object "kube-system"/"original-pull-secret" not registered Oct 31 12:31:20.562519 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.562490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:20.562684 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.562619 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 12:31:20.562684 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.562653 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 12:31:20.562684 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.562667 2580 projected.go:196] Error preparing data for projected volume kube-api-access-c4sk5 for pod openshift-network-diagnostics/network-check-target-gjvxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:20.562779 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.562770 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5 podName:f05d824e-3be1-46f6-8df0-2ade9675e534 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:36.562732169 +0000 UTC m=+33.360170230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c4sk5" (UniqueName: "kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5") pod "network-check-target-gjvxs" (UID: "f05d824e-3be1-46f6-8df0-2ade9675e534") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 12:31:20.689445 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.689423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:20.689555 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.689464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:20.689555 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.689540 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:20.689675 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.689645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:20.689783 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.689751 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:20.689912 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:20.689886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:20.882453 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.882226 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"f26c2157d0c92d7dbe9fb10a9e283e8d9e838626b0096e1af1d61a7d26ca0b37"} Oct 31 12:31:20.882913 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.882516 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"7b0eb8f3d75e360c8eb9a00e30a10a053b5930903b8c86221c368d26d539000c"} Oct 31 12:31:20.882913 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.882563 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"d0b8b2ae44ad268956fba0b92c587c36bab7283e6ed473030f3eb934a7ba25e8"} Oct 31 12:31:20.882913 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:20.882607 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"853de13b32771ae48988da83599751660cd19395b220f0f6ce250959861a7543"} Oct 31 12:31:21.887159 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:21.887130 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"f3a28ba994e427d0e3d5deb059e5936fb22bb9a700187e1dfa9d9c7f6595a858"} Oct 31 12:31:21.887589 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:21.887196 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"e3ba9a864d46bd43d0e37353d30edcdabbce15bf0b8d4c6376c2fcb7a636b74e"} Oct 31 12:31:22.689400 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:22.689374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:22.689400 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:22.689394 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:22.689671 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:22.689374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:22.689671 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:22.689470 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:22.689671 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:22.689570 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:22.689787 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:22.689676 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:22.891450 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:22.891414 2580 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="c81841a1450b9168278a6504581c2bd88ad7ac6e890a53cfe8c07f0d3631ede3" exitCode=0 Oct 31 12:31:22.891450 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:22.891453 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerDied","Data":"c81841a1450b9168278a6504581c2bd88ad7ac6e890a53cfe8c07f0d3631ede3"} Oct 31 12:31:23.896931 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:23.896897 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"2d8536fafa929b8bb591a729c7fe40b3713c742e366c186514ed5f92f90114c6"} Oct 31 12:31:23.899622 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:23.899593 2580 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="6230b924d6c7d1583743bbb3775310712d7b2e70b911fdc157a03b6c3579ce51" exitCode=0 Oct 31 12:31:23.899753 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:23.899649 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerDied","Data":"6230b924d6c7d1583743bbb3775310712d7b2e70b911fdc157a03b6c3579ce51"} Oct 31 12:31:24.689580 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:24.689549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:24.689743 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:24.689716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:24.689743 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:24.689726 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:24.689743 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:24.689726 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:24.689873 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:24.689788 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:24.689914 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:24.689868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:24.904499 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:24.904458 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" event={"ID":"59030007-53c1-42c9-baec-a4d2ec90f3dd","Type":"ContainerStarted","Data":"a22e561c5096ad5af8d7ede3971b665d1deab07105350d4baec057b7c670c8c1"} Oct 31 12:31:24.931231 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:24.931189 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9tkzk" podStartSLOduration=8.146609823 podStartE2EDuration="20.931177203s" totalImagesPullingTime="12.78456738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:24.929625402 +0000 UTC m=+21.727063469" watchObservedRunningTime="2026-10-31 12:31:24.931177203 +0000 UTC m=+21.728615268" Oct 31 12:31:25.909960 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:25.909916 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" event={"ID":"4a8a0aad-3971-455a-8fda-f40c28d8bc13","Type":"ContainerStarted","Data":"b2e94bc665ac5a653eb79b9113c82cf1e14d830902ad20742f091fb13fe6f6c1"} Oct 31 12:31:25.944472 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:25.944432 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" podStartSLOduration=6.381310931 podStartE2EDuration="21.944419354s" totalImagesPullingTime="15.563108423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:25.942893141 +0000 UTC m=+22.740331206" watchObservedRunningTime="2026-10-31 12:31:25.944419354 +0000 UTC m=+22.741857419" Oct 31 12:31:26.689346 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.689319 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:26.689537 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.689366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:26.689537 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.689315 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:26.689537 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:26.689425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:26.689677 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:26.689534 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:26.689677 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:26.689624 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:26.768417 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.768388 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sk7rm"] Oct 31 12:31:26.772136 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.772111 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gjvxs"] Oct 31 12:31:26.772253 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.772167 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kl4n"] Oct 31 12:31:26.912119 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.912091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:26.912119 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.912091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:26.912119 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.912102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:26.912664 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.912464 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:26.912664 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:26.912473 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:26.912664 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:26.912557 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:26.912664 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.912595 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:26.912664 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:26.912590 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:26.912891 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.912673 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:26.930038 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.930009 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:26.932329 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:26.930236 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:28.688805 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:28.688771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:28.688805 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:28.688803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:28.689345 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:28.688863 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:28.689345 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:28.688878 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:28.689345 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:28.688928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:28.689345 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:28.689012 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:30.688837 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:30.688808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:30.689280 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:30.688936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:30.689280 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:30.688947 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:30.689280 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:30.688963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:30.689280 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:30.689016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:30.689280 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:30.689087 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:32.688899 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:32.688857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:32.689302 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:32.688921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:32.689302 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:32.689017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:32.689302 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:32.689016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk7rm" podUID="8cea9a4e-e914-40b6-9237-345be4aff28c" Oct 31 12:31:32.689302 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:32.689102 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kl4n" podUID="fb0b76f8-eb8d-45ee-afec-b393223b0504" Oct 31 12:31:32.689302 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:32.689173 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gjvxs" podUID="f05d824e-3be1-46f6-8df0-2ade9675e534" Oct 31 12:31:33.514739 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.514708 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeReady" Oct 31 12:31:33.514882 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.514820 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 12:31:33.551025 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.550999 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6ps96"] Oct 31 12:31:33.591755 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.591725 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tn6pm"] Oct 31 12:31:33.591928 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.591763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6ps96" Oct 31 12:31:33.600381 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.600346 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.601275 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.601248 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.605020 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.604698 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxg8d"] Oct 31 12:31:33.606541 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.606512 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" Oct 31 12:31:33.613445 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.613425 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4"] Oct 31 12:31:33.613554 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.613538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" Oct 31 12:31:33.617022 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.617005 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc"] Oct 31 12:31:33.617160 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.617140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" Oct 31 12:31:33.621842 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.621826 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-r82tg\"" Oct 31 12:31:33.621959 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.621828 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7vnxg"] Oct 31 12:31:33.621959 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.621893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.626458 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.626444 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 12:31:33.626598 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.626581 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nwqs2\"" Oct 31 12:31:33.636968 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.636948 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2"] Oct 31 12:31:33.637071 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.637020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.659720 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.659701 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tc2tp"] Oct 31 12:31:33.659840 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.659822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.662950 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.662930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgbcf\" (UniqueName: \"kubernetes.io/projected/5b057b6d-b9d1-4642-95f3-41ba5498c24d-kube-api-access-fgbcf\") pod \"kas-connection-checker-685895c7c9-6ps96\" (UID: \"5b057b6d-b9d1-4642-95f3-41ba5498c24d\") " pod="kube-system/kas-connection-checker-685895c7c9-6ps96" Oct 31 12:31:33.672900 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.672875 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9"] Oct 31 12:31:33.672993 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.672960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.683902 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.683884 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-865bf4d8bc-d5gwq"] Oct 31 12:31:33.684006 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.683992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.695596 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.695576 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pnzjp"] Oct 31 12:31:33.695918 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.695690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.696279 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696258 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.696456 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696267 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.696456 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696350 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.696456 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696359 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 12:31:33.696456 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696389 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.696456 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696394 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 12:31:33.696456 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696446 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.696456 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696459 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696487 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696504 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696461 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696533 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696535 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696546 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696549 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.696861 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.696566 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.704236 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.704210 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 12:31:33.705845 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.705817 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 12:31:33.706887 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.706869 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 12:31:33.709767 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.709750 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f8479bcff-g6cvb"] Oct 31 12:31:33.709875 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.709859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:33.722016 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.721994 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 12:31:33.722016 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722009 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 12:31:33.722016 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722016 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 12:31:33.722202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722042 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 12:31:33.722202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722096 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 12:31:33.722202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722124 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vbv96\"" Oct 31 12:31:33.722202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722133 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9dfhb\"" Oct 31 12:31:33.722202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722156 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 12:31:33.722202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722160 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5tdjb\"" Oct 31 12:31:33.722202 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722175 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 12:31:33.722457 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722250 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 12:31:33.722457 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722278 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-29qxx\"" Oct 31 12:31:33.722457 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722291 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pnqgm\"" Oct 31 12:31:33.722457 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.722308 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mdmxf\"" Oct 31 12:31:33.730973 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.730955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.734933 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.734916 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8"] Oct 31 12:31:33.751029 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.751009 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85dd98c8f4-j6vtq"] Oct 31 12:31:33.751127 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.751112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.763377 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763356 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.763484 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tr4\" (UniqueName: \"kubernetes.io/projected/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d-kube-api-access-s7tr4\") pod \"kas-connection-checker-685895c7c9-tn6pm\" (UID: \"e8d6247b-578f-4ee3-8fb4-01bb7da39a7d\") " pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" Oct 31 12:31:33.763484 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.763556 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9r6x\" (UniqueName: \"kubernetes.io/projected/55bcf0cc-7eef-4cbe-9d72-140932580c56-kube-api-access-p9r6x\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.763556 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bknk7\" (UniqueName: \"kubernetes.io/projected/719b7f16-3b6a-402e-8301-7b498a9ad131-kube-api-access-bknk7\") pod \"volume-data-source-validator-58c9f5f46d-sxrj4\" (UID: \"719b7f16-3b6a-402e-8301-7b498a9ad131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" Oct 31 12:31:33.763556 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khg96\" (UniqueName: \"kubernetes.io/projected/a8053559-57b0-468c-a369-893764b0ed4d-kube-api-access-khg96\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.763660 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.763660 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-tmp\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.763724 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bcf0cc-7eef-4cbe-9d72-140932580c56-serving-cert\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.763724 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.763784 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.763784 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763753 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmcf\" (UniqueName: \"kubernetes.io/projected/579b4046-76c9-4d7e-b77a-9888510a22a8-kube-api-access-8pmcf\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.763784 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.763866 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236577a8-0720-452b-83fa-986cb9aeba14-serving-cert\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.763866 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxwp\" (UniqueName: \"kubernetes.io/projected/746d043e-1f60-464f-ba50-6a437eb9089f-kube-api-access-6dxwp\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.763925 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2fm\" (UniqueName: \"kubernetes.io/projected/236577a8-0720-452b-83fa-986cb9aeba14-kube-api-access-mq2fm\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.763925 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.763984 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236577a8-0720-452b-83fa-986cb9aeba14-config\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.764012 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.763992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvv9r\" (UniqueName: \"kubernetes.io/projected/dee0dda1-07c9-43fc-b7d6-f09b502d0149-kube-api-access-nvv9r\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.764044 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-snapshots\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.764044 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-stats-auth\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.764100 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-default-certificate\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.764100 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.764165 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dee0dda1-07c9-43fc-b7d6-f09b502d0149-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.764207 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ppx\" (UniqueName: \"kubernetes.io/projected/c765c0f2-1bc6-4eee-9fde-317749984f5c-kube-api-access-w2ppx\") pod \"kas-connection-checker-685895c7c9-hxg8d\" (UID: \"c765c0f2-1bc6-4eee-9fde-317749984f5c\") " pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" Oct 31 12:31:33.764239 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgbcf\" (UniqueName: \"kubernetes.io/projected/5b057b6d-b9d1-4642-95f3-41ba5498c24d-kube-api-access-fgbcf\") pod \"kas-connection-checker-685895c7c9-6ps96\" (UID: \"5b057b6d-b9d1-4642-95f3-41ba5498c24d\") " pod="kube-system/kas-connection-checker-685895c7c9-6ps96" Oct 31 12:31:33.764272 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-config\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.764272 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.764261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8053559-57b0-468c-a369-893764b0ed4d-serving-cert\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.765279 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.765227 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5"] Oct 31 12:31:33.765356 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.765339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.776852 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.776823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgbcf\" (UniqueName: \"kubernetes.io/projected/5b057b6d-b9d1-4642-95f3-41ba5498c24d-kube-api-access-fgbcf\") pod \"kas-connection-checker-685895c7c9-6ps96\" (UID: \"5b057b6d-b9d1-4642-95f3-41ba5498c24d\") " pod="kube-system/kas-connection-checker-685895c7c9-6ps96" Oct 31 12:31:33.777546 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.777528 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6ps96"] Oct 31 12:31:33.777628 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.777556 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p4psl"] Oct 31 12:31:33.777689 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.777659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" Oct 31 12:31:33.791765 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791743 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4"] Oct 31 12:31:33.791849 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791769 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxg8d"] Oct 31 12:31:33.791849 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791782 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tn6pm"] Oct 31 12:31:33.791849 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791835 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc"] Oct 31 12:31:33.791954 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791851 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pnzjp"] Oct 31 12:31:33.791954 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791866 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2"] Oct 31 12:31:33.791954 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791882 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-865bf4d8bc-d5gwq"] Oct 31 12:31:33.791954 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791893 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8"] Oct 31 12:31:33.791954 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.791912 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bqcvc"] Oct 31 12:31:33.792095 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.792032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.797668 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.797651 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 12:31:33.797856 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.797843 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.798013 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.797998 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.798216 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.798204 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 12:31:33.802312 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.802290 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.802395 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.802307 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.802395 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.802322 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:31:33.802510 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.802405 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 12:31:33.802510 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.802415 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.802907 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.802891 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.802907 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.802890 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 12:31:33.807979 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.807961 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85dd98c8f4-j6vtq"] Oct 31 12:31:33.808068 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.807994 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tc2tp"] Oct 31 12:31:33.808068 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.808017 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7vnxg"] Oct 31 12:31:33.808068 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.808037 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4psl"] Oct 31 12:31:33.808068 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.808057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:33.808232 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.808062 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9"] Oct 31 12:31:33.808232 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.808208 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5"] Oct 31 12:31:33.808310 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.808259 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bqcvc"] Oct 31 12:31:33.808310 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.808279 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8479bcff-g6cvb"] Oct 31 12:31:33.822159 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822137 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 12:31:33.822270 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822257 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 12:31:33.822430 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822417 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 12:31:33.822460 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822442 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nc6nm\"" Oct 31 12:31:33.822518 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822508 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:31:33.822554 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822548 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:31:33.822591 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822554 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cclp9\"" Oct 31 12:31:33.822591 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822559 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nthx6\"" Oct 31 12:31:33.822591 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822564 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 12:31:33.822697 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822605 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrn76\"" Oct 31 12:31:33.822727 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.822697 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xwjnx\"" Oct 31 12:31:33.864935 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.864909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-snapshots\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.864935 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.864936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvv9r\" (UniqueName: \"kubernetes.io/projected/dee0dda1-07c9-43fc-b7d6-f09b502d0149-kube-api-access-nvv9r\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.865091 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.864953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-default-certificate\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.865091 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.864971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fee2a93c-fe10-47e5-840c-8d8be03b2715-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.865091 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.864989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dee0dda1-07c9-43fc-b7d6-f09b502d0149-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.865091 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ppx\" (UniqueName: \"kubernetes.io/projected/c765c0f2-1bc6-4eee-9fde-317749984f5c-kube-api-access-w2ppx\") pod \"kas-connection-checker-685895c7c9-hxg8d\" (UID: \"c765c0f2-1bc6-4eee-9fde-317749984f5c\") " pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" Oct 31 12:31:33.865091 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.865299 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.865299 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-config\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.865299 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgq6\" (UniqueName: \"kubernetes.io/projected/8c751f98-9133-4738-9052-8dba6020cda6-kube-api-access-skgq6\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.865299 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r6x\" (UniqueName: \"kubernetes.io/projected/55bcf0cc-7eef-4cbe-9d72-140932580c56-kube-api-access-p9r6x\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.865473 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.865473 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-tmp\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.865473 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c751f98-9133-4738-9052-8dba6020cda6-tmp-dir\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.865542 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.865597 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.365580232 +0000 UTC m=+31.163018289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:33.865672 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxwp\" (UniqueName: \"kubernetes.io/projected/746d043e-1f60-464f-ba50-6a437eb9089f-kube-api-access-6dxwp\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865680 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-snapshots\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fm\" (UniqueName: \"kubernetes.io/projected/236577a8-0720-452b-83fa-986cb9aeba14-kube-api-access-mq2fm\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-stats-auth\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.865790 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.865843 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls podName:746d043e-1f60-464f-ba50-6a437eb9089f nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.365828544 +0000 UTC m=+31.163266601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sckc2" (UID: "746d043e-1f60-464f-ba50-6a437eb9089f") : secret "samples-operator-tls" not found Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.865955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-config\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866003 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dee0dda1-07c9-43fc-b7d6-f09b502d0149-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866008 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.866061 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:33.866080 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866066 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-tmp\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.866097 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.366085511 +0000 UTC m=+31.163523556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236577a8-0720-452b-83fa-986cb9aeba14-config\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/698d7a34-d950-442b-b12e-b0954c8fe873-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5nm4\" (UniqueName: \"kubernetes.io/projected/bf409f2b-9f95-4f84-bbd6-d9cc841fc074-kube-api-access-j5nm4\") pod \"network-check-source-6f78d59f99-62jh5\" (UID: \"bf409f2b-9f95-4f84-bbd6-d9cc841fc074\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bknk7\" (UniqueName: \"kubernetes.io/projected/719b7f16-3b6a-402e-8301-7b498a9ad131-kube-api-access-bknk7\") pod \"volume-data-source-validator-58c9f5f46d-sxrj4\" (UID: \"719b7f16-3b6a-402e-8301-7b498a9ad131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.866353 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.366339637 +0000 UTC m=+31.163777704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fee2a93c-fe10-47e5-840c-8d8be03b2715-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.866573 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tr4\" (UniqueName: \"kubernetes.io/projected/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d-kube-api-access-s7tr4\") pod \"kas-connection-checker-685895c7c9-tn6pm\" (UID: \"e8d6247b-578f-4ee3-8fb4-01bb7da39a7d\") " pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866713 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236577a8-0720-452b-83fa-986cb9aeba14-config\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8053559-57b0-468c-a369-893764b0ed4d-serving-cert\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866737 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.866968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqrdr\" (UniqueName: \"kubernetes.io/projected/fee2a93c-fe10-47e5-840c-8d8be03b2715-kube-api-access-zqrdr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khg96\" (UniqueName: \"kubernetes.io/projected/a8053559-57b0-468c-a369-893764b0ed4d-kube-api-access-khg96\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bcf0cc-7eef-4cbe-9d72-140932580c56-serving-cert\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.867226 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.867962 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmcf\" (UniqueName: \"kubernetes.io/projected/579b4046-76c9-4d7e-b77a-9888510a22a8-kube-api-access-8pmcf\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.867962 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c751f98-9133-4738-9052-8dba6020cda6-config-volume\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.867962 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntcw7\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.867962 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236577a8-0720-452b-83fa-986cb9aeba14-serving-cert\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.867962 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.867440 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.868194 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.868030 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-default-certificate\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.868194 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.868031 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.868788 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.868765 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-stats-auth\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.869285 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.869262 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8053559-57b0-468c-a369-893764b0ed4d-serving-cert\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.869748 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.869725 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236577a8-0720-452b-83fa-986cb9aeba14-serving-cert\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.869748 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.869746 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bcf0cc-7eef-4cbe-9d72-140932580c56-serving-cert\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.899238 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.899215 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tr4\" (UniqueName: \"kubernetes.io/projected/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d-kube-api-access-s7tr4\") pod \"kas-connection-checker-685895c7c9-tn6pm\" (UID: \"e8d6247b-578f-4ee3-8fb4-01bb7da39a7d\") " pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" Oct 31 12:31:33.900180 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.900048 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:31:33.900180 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.900151 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:31:33.901095 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.901071 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khg96\" (UniqueName: \"kubernetes.io/projected/a8053559-57b0-468c-a369-893764b0ed4d-kube-api-access-khg96\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.902033 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.902004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r6x\" (UniqueName: \"kubernetes.io/projected/55bcf0cc-7eef-4cbe-9d72-140932580c56-kube-api-access-p9r6x\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:33.902375 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.902337 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ppx\" (UniqueName: \"kubernetes.io/projected/c765c0f2-1bc6-4eee-9fde-317749984f5c-kube-api-access-w2ppx\") pod \"kas-connection-checker-685895c7c9-hxg8d\" (UID: \"c765c0f2-1bc6-4eee-9fde-317749984f5c\") " pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" Oct 31 12:31:33.902523 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.902502 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2fm\" (UniqueName: \"kubernetes.io/projected/236577a8-0720-452b-83fa-986cb9aeba14-kube-api-access-mq2fm\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:33.902845 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.902822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxwp\" (UniqueName: \"kubernetes.io/projected/746d043e-1f60-464f-ba50-6a437eb9089f-kube-api-access-6dxwp\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:33.903242 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.903220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvv9r\" (UniqueName: \"kubernetes.io/projected/dee0dda1-07c9-43fc-b7d6-f09b502d0149-kube-api-access-nvv9r\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:33.903363 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.903305 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmcf\" (UniqueName: \"kubernetes.io/projected/579b4046-76c9-4d7e-b77a-9888510a22a8-kube-api-access-8pmcf\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:33.904424 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.904399 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bknk7\" (UniqueName: \"kubernetes.io/projected/719b7f16-3b6a-402e-8301-7b498a9ad131-kube-api-access-bknk7\") pod \"volume-data-source-validator-58c9f5f46d-sxrj4\" (UID: \"719b7f16-3b6a-402e-8301-7b498a9ad131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" Oct 31 12:31:33.918547 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.918488 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6ps96" Oct 31 12:31:33.940398 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.940377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" Oct 31 12:31:33.954190 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.954170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" Oct 31 12:31:33.968021 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.967988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.968129 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.968129 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.968129 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.968129 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrdr\" (UniqueName: \"kubernetes.io/projected/fee2a93c-fe10-47e5-840c-8d8be03b2715-kube-api-access-zqrdr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.968364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.968364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.968364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c751f98-9133-4738-9052-8dba6020cda6-config-volume\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.968364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntcw7\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.968364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjsf\" (UniqueName: \"kubernetes.io/projected/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-kube-api-access-zxjsf\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:33.968364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fee2a93c-fe10-47e5-840c-8d8be03b2715-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.968364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.968706 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-cert\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:33.968706 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgq6\" (UniqueName: \"kubernetes.io/projected/8c751f98-9133-4738-9052-8dba6020cda6-kube-api-access-skgq6\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.968706 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.968454 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:33.968706 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.968484 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:33.968706 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.968540 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.468520927 +0000 UTC m=+31.265958993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:33.968949 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c751f98-9133-4738-9052-8dba6020cda6-tmp-dir\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.968949 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.968949 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968748 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c751f98-9133-4738-9052-8dba6020cda6-tmp-dir\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.968949 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.968949 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.968949 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.968997 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/698d7a34-d950-442b-b12e-b0954c8fe873-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5nm4\" (UniqueName: \"kubernetes.io/projected/bf409f2b-9f95-4f84-bbd6-d9cc841fc074-kube-api-access-j5nm4\") pod \"network-check-source-6f78d59f99-62jh5\" (UID: \"bf409f2b-9f95-4f84-bbd6-d9cc841fc074\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fee2a93c-fe10-47e5-840c-8d8be03b2715-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.969244 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.969776 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969261 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c751f98-9133-4738-9052-8dba6020cda6-config-volume\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:33.969776 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.969340 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:33.969776 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.969353 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:33.969776 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.969396 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.469381102 +0000 UTC m=+31.266819162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:33.969985 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.969985 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.969893 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.969985 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.969937 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:31:33.970121 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.969988 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls podName:8c751f98-9133-4738-9052-8dba6020cda6 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.469971892 +0000 UTC m=+31.267410063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls") pod "dns-default-p4psl" (UID: "8c751f98-9133-4738-9052-8dba6020cda6") : secret "dns-default-metrics-tls" not found Oct 31 12:31:33.970208 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.970189 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:33.970283 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:33.970271 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:34.470230369 +0000 UTC m=+31.267668431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:33.970723 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.970697 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/698d7a34-d950-442b-b12e-b0954c8fe873-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:33.970944 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.970919 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.971448 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.971424 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fee2a93c-fe10-47e5-840c-8d8be03b2715-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.971538 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.971499 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.971758 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.971733 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.971841 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.971797 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.972473 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.972435 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fee2a93c-fe10-47e5-840c-8d8be03b2715-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.972556 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.972506 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.973247 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.973197 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.973311 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.973295 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.974922 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.974896 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.985979 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.985950 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.985979 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.985965 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrdr\" (UniqueName: \"kubernetes.io/projected/fee2a93c-fe10-47e5-840c-8d8be03b2715-kube-api-access-zqrdr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:33.986892 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.986870 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntcw7\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.988354 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.988328 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5nm4\" (UniqueName: \"kubernetes.io/projected/bf409f2b-9f95-4f84-bbd6-d9cc841fc074-kube-api-access-j5nm4\") pod \"network-check-source-6f78d59f99-62jh5\" (UID: \"bf409f2b-9f95-4f84-bbd6-d9cc841fc074\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" Oct 31 12:31:33.992607 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.992577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:33.994878 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.994842 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:33.995647 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.995612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:33.996577 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:33.996557 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgq6\" (UniqueName: \"kubernetes.io/projected/8c751f98-9133-4738-9052-8dba6020cda6-kube-api-access-skgq6\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:34.003990 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.003828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" Oct 31 12:31:34.043802 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.043517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:34.058262 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.058078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:34.070474 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.069819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjsf\" (UniqueName: \"kubernetes.io/projected/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-kube-api-access-zxjsf\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:34.070474 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.069871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-cert\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:34.074197 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.073967 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-cert\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:34.085686 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.085619 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjsf\" (UniqueName: \"kubernetes.io/projected/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-kube-api-access-zxjsf\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:34.134612 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.134584 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:34.179272 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.179042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" Oct 31 12:31:34.239148 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.238978 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:34.267500 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.267464 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tn6pm"] Oct 31 12:31:34.270366 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.270342 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6ps96"] Oct 31 12:31:34.271312 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.271290 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxg8d"] Oct 31 12:31:34.272109 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:34.272087 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d6247b_578f_4ee3_8fb4_01bb7da39a7d.slice/crio-979f00a02414c60112e49ee9012cba264a0369fa8653a7e48489ef1c0200110c WatchSource:0}: Error finding container 979f00a02414c60112e49ee9012cba264a0369fa8653a7e48489ef1c0200110c: Status 404 returned error can't find the container with id 979f00a02414c60112e49ee9012cba264a0369fa8653a7e48489ef1c0200110c Oct 31 12:31:34.272801 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:34.272783 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc765c0f2_1bc6_4eee_9fde_317749984f5c.slice/crio-7cfad452ea40a72182fc93d112e4ffeb0236974801cb84dbe6bd42b96e5a8ea5 WatchSource:0}: Error finding container 7cfad452ea40a72182fc93d112e4ffeb0236974801cb84dbe6bd42b96e5a8ea5: Status 404 returned error can't find the container with id 7cfad452ea40a72182fc93d112e4ffeb0236974801cb84dbe6bd42b96e5a8ea5 Oct 31 12:31:34.274618 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:34.274581 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b057b6d_b9d1_4642_95f3_41ba5498c24d.slice/crio-d7883d6e70ed6e5d818f140fd7b5963430ab4d60a23a9073ede8f24eedcf7cb2 WatchSource:0}: Error finding container d7883d6e70ed6e5d818f140fd7b5963430ab4d60a23a9073ede8f24eedcf7cb2: Status 404 returned error can't find the container with id d7883d6e70ed6e5d818f140fd7b5963430ab4d60a23a9073ede8f24eedcf7cb2 Oct 31 12:31:34.330171 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.326718 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4"] Oct 31 12:31:34.331264 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.331239 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7vnxg"] Oct 31 12:31:34.347130 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.347046 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tc2tp"] Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.372589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.372652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.372681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.372741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.372957 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.37293827 +0000 UTC m=+32.170376333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.373044 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.373082 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.373071693 +0000 UTC m=+32.170509737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.373135 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.373165 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.373155647 +0000 UTC m=+32.170593708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.373218 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 12:31:34.373274 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.373247 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls podName:746d043e-1f60-464f-ba50-6a437eb9089f nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.373238185 +0000 UTC m=+32.170676448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sckc2" (UID: "746d043e-1f60-464f-ba50-6a437eb9089f") : secret "samples-operator-tls" not found Oct 31 12:31:34.376713 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.376325 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9"] Oct 31 12:31:34.376863 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:34.376833 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236577a8_0720_452b_83fa_986cb9aeba14.slice/crio-8514d431fc512977b1a1605881be8be2bb1b917cfe73bf7e8e3aca8209503a60 WatchSource:0}: Error finding container 8514d431fc512977b1a1605881be8be2bb1b917cfe73bf7e8e3aca8209503a60: Status 404 returned error can't find the container with id 8514d431fc512977b1a1605881be8be2bb1b917cfe73bf7e8e3aca8209503a60 Oct 31 12:31:34.413237 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.413213 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8"] Oct 31 12:31:34.417740 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:34.417717 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee2a93c_fe10_47e5_840c_8d8be03b2715.slice/crio-4a2660e80bf82352c5db70044254f189f0032d30f8ca9d91d95c670a2bdb160e WatchSource:0}: Error finding container 4a2660e80bf82352c5db70044254f189f0032d30f8ca9d91d95c670a2bdb160e: Status 404 returned error can't find the container with id 4a2660e80bf82352c5db70044254f189f0032d30f8ca9d91d95c670a2bdb160e Oct 31 12:31:34.472310 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.472285 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5"] Oct 31 12:31:34.473344 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.473321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:34.473454 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.473367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:34.473454 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.473422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:34.473556 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.473493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:34.473556 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473500 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:34.473556 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473520 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:34.473556 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473539 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:31:34.473745 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473582 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.473562658 +0000 UTC m=+32.271000716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:34.473745 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473611 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:34.473745 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473651 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:34.473745 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473659 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls podName:8c751f98-9133-4738-9052-8dba6020cda6 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.473649654 +0000 UTC m=+32.271087699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls") pod "dns-default-p4psl" (UID: "8c751f98-9133-4738-9052-8dba6020cda6") : secret "dns-default-metrics-tls" not found Oct 31 12:31:34.473745 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473664 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:34.473745 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473675 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.473668356 +0000 UTC m=+32.271106400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:34.473745 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:34.473708 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:35.473692833 +0000 UTC m=+32.271130883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:34.474659 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:34.474615 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf409f2b_9f95_4f84_bbd6_d9cc841fc074.slice/crio-034cd7c334403750b4defa906b0e09b93223e3e3d5b1a17ec4691b6a2cce323c WatchSource:0}: Error finding container 034cd7c334403750b4defa906b0e09b93223e3e3d5b1a17ec4691b6a2cce323c: Status 404 returned error can't find the container with id 034cd7c334403750b4defa906b0e09b93223e3e3d5b1a17ec4691b6a2cce323c Oct 31 12:31:34.561651 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.561566 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bqcvc"] Oct 31 12:31:34.564762 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:34.564735 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode040a356_8196_40c6_8f2c_a94bb4c6bfe4.slice/crio-a27a74f42953674491bf7a41068040af323bca7609f05ef24d11251e595822f5 WatchSource:0}: Error finding container a27a74f42953674491bf7a41068040af323bca7609f05ef24d11251e595822f5: Status 404 returned error can't find the container with id a27a74f42953674491bf7a41068040af323bca7609f05ef24d11251e595822f5 Oct 31 12:31:34.688713 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.688683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:34.688855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.688750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:34.688855 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.688780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:34.700994 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.700974 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:31:34.701318 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.701001 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:31:34.701318 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.701012 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:31:34.701318 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.701070 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:31:34.701318 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.701125 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:31:34.930687 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.930595 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" event={"ID":"bf409f2b-9f95-4f84-bbd6-d9cc841fc074","Type":"ContainerStarted","Data":"034cd7c334403750b4defa906b0e09b93223e3e3d5b1a17ec4691b6a2cce323c"} Oct 31 12:31:34.931767 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.931745 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" event={"ID":"236577a8-0720-452b-83fa-986cb9aeba14","Type":"ContainerStarted","Data":"8514d431fc512977b1a1605881be8be2bb1b917cfe73bf7e8e3aca8209503a60"} Oct 31 12:31:34.932887 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.932850 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" event={"ID":"55bcf0cc-7eef-4cbe-9d72-140932580c56","Type":"ContainerStarted","Data":"34b30d9b6d3b64aa18e694b99251502e5d7fffc7510f82a6415edd2a779ceccd"} Oct 31 12:31:34.933902 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.933884 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" event={"ID":"fee2a93c-fe10-47e5-840c-8d8be03b2715","Type":"ContainerStarted","Data":"4a2660e80bf82352c5db70044254f189f0032d30f8ca9d91d95c670a2bdb160e"} Oct 31 12:31:34.935322 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.935300 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6ps96" event={"ID":"5b057b6d-b9d1-4642-95f3-41ba5498c24d","Type":"ContainerStarted","Data":"70f423f51a9f500aa6352905959ca9ec7d3afce2598105d0d72ededd1e913433"} Oct 31 12:31:34.935412 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.935333 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6ps96" event={"ID":"5b057b6d-b9d1-4642-95f3-41ba5498c24d","Type":"ContainerStarted","Data":"d7883d6e70ed6e5d818f140fd7b5963430ab4d60a23a9073ede8f24eedcf7cb2"} Oct 31 12:31:34.936355 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.936335 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bqcvc" event={"ID":"e040a356-8196-40c6-8f2c-a94bb4c6bfe4","Type":"ContainerStarted","Data":"a27a74f42953674491bf7a41068040af323bca7609f05ef24d11251e595822f5"} Oct 31 12:31:34.937408 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.937386 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" event={"ID":"719b7f16-3b6a-402e-8301-7b498a9ad131","Type":"ContainerStarted","Data":"c45f4aaf555deb61468e74093c2c65be2edcaea0313d9ac73aee2f7a7de02d60"} Oct 31 12:31:34.938452 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.938433 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" event={"ID":"a8053559-57b0-468c-a369-893764b0ed4d","Type":"ContainerStarted","Data":"571d12f96a06f1644c4d7dc77e1863e14c5570d657c52f9b2e41f8bad45d3aed"} Oct 31 12:31:34.940237 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.940194 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" event={"ID":"c765c0f2-1bc6-4eee-9fde-317749984f5c","Type":"ContainerStarted","Data":"42968b35f39d663e3a24609f0201392bec8b29daf50d76a206557b8eb21934fc"} Oct 31 12:31:34.940330 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.940287 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" event={"ID":"c765c0f2-1bc6-4eee-9fde-317749984f5c","Type":"ContainerStarted","Data":"7cfad452ea40a72182fc93d112e4ffeb0236974801cb84dbe6bd42b96e5a8ea5"} Oct 31 12:31:34.944840 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.944818 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" event={"ID":"e8d6247b-578f-4ee3-8fb4-01bb7da39a7d","Type":"ContainerStarted","Data":"2ae7591f0a5cc0f5399d4ee23118f17aeeccbab82255c5cb6fe4c31bff232ad3"} Oct 31 12:31:34.944902 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.944850 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" event={"ID":"e8d6247b-578f-4ee3-8fb4-01bb7da39a7d","Type":"ContainerStarted","Data":"979f00a02414c60112e49ee9012cba264a0369fa8653a7e48489ef1c0200110c"} Oct 31 12:31:34.953948 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.953900 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6ps96" podStartSLOduration=381.953889228 podStartE2EDuration="6m21.953889228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:34.953081218 +0000 UTC m=+31.750519285" watchObservedRunningTime="2026-10-31 12:31:34.953889228 +0000 UTC m=+31.751327294" Oct 31 12:31:34.971706 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.971672 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" podStartSLOduration=381.97162785 podStartE2EDuration="6m21.97162785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:34.971420265 +0000 UTC m=+31.768858335" watchObservedRunningTime="2026-10-31 12:31:34.97162785 +0000 UTC m=+31.769065915" Oct 31 12:31:34.989956 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:34.989913 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" podStartSLOduration=381.989902502 podStartE2EDuration="6m21.989902502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:34.98930514 +0000 UTC m=+31.786743198" watchObservedRunningTime="2026-10-31 12:31:34.989902502 +0000 UTC m=+31.787340568" Oct 31 12:31:35.382848 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.382806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:35.383008 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.382870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:35.383008 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.382900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:35.383131 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.383070 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:35.383168 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.383138 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.383118688 +0000 UTC m=+34.180556738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:35.383718 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.383700 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:35.383798 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.383762 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.38374407 +0000 UTC m=+34.181182125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:35.383956 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.383931 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 12:31:35.384006 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.383999 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls podName:746d043e-1f60-464f-ba50-6a437eb9089f nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.38398158 +0000 UTC m=+34.181419639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sckc2" (UID: "746d043e-1f60-464f-ba50-6a437eb9089f") : secret "samples-operator-tls" not found Oct 31 12:31:35.384063 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.383938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:35.384063 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.384025 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.384012248 +0000 UTC m=+34.181450316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:35.485115 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.485073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:35.485278 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.485137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:35.485278 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485252 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:35.485278 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485266 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:31:35.485278 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485276 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:35.485433 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485322 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls podName:8c751f98-9133-4738-9052-8dba6020cda6 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.485305647 +0000 UTC m=+34.282743706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls") pod "dns-default-p4psl" (UID: "8c751f98-9133-4738-9052-8dba6020cda6") : secret "dns-default-metrics-tls" not found Oct 31 12:31:35.485433 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485341 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.485331916 +0000 UTC m=+34.282769962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:35.485433 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.485359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:35.485587 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:35.485430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:35.485587 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485542 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:35.485587 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485564 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:35.485587 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485575 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:35.485821 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485596 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.485579527 +0000 UTC m=+34.283017571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:35.485821 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:35.485617 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:37.485606323 +0000 UTC m=+34.283044370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:36.497948 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.496977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:36.497948 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.497099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:36.497948 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:36.497243 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 12:31:36.497948 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:36.497303 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs podName:fb0b76f8-eb8d-45ee-afec-b393223b0504 nodeName:}" failed. No retries permitted until 2026-10-31 12:32:08.497284497 +0000 UTC m=+65.294722547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs") pod "network-metrics-daemon-5kl4n" (UID: "fb0b76f8-eb8d-45ee-afec-b393223b0504") : secret "metrics-daemon-secret" not found Oct 31 12:31:36.500549 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.500526 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:36.586744 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.586709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:36.598231 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.598202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:36.601237 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.601209 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:36.869868 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.869833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:36.928737 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.928699 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sk7rm"] Oct 31 12:31:36.934892 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:36.934135 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cea9a4e_e914_40b6_9237_345be4aff28c.slice/crio-fa6cc5e19d492cf2cb49b2106f61c27e5ff4c0d9dc01b6639dc114076e24f040 WatchSource:0}: Error finding container fa6cc5e19d492cf2cb49b2106f61c27e5ff4c0d9dc01b6639dc114076e24f040: Status 404 returned error can't find the container with id fa6cc5e19d492cf2cb49b2106f61c27e5ff4c0d9dc01b6639dc114076e24f040 Oct 31 12:31:36.959482 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.959447 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" event={"ID":"719b7f16-3b6a-402e-8301-7b498a9ad131","Type":"ContainerStarted","Data":"a61776d5dab53704fff7a23068755ceec31710a4d8faa043174eeccc6ce27e28"} Oct 31 12:31:36.961972 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.961947 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sk7rm" event={"ID":"8cea9a4e-e914-40b6-9237-345be4aff28c","Type":"ContainerStarted","Data":"fa6cc5e19d492cf2cb49b2106f61c27e5ff4c0d9dc01b6639dc114076e24f040"} Oct 31 12:31:36.978490 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:36.978430 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" podStartSLOduration=345.687704195 podStartE2EDuration="5m47.978413824s" totalImagesPullingTime="2.290709629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:36.977284479 +0000 UTC m=+33.774722544" watchObservedRunningTime="2026-10-31 12:31:36.978413824 +0000 UTC m=+33.775851891" Oct 31 12:31:37.175714 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.175539 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gjvxs"] Oct 31 12:31:37.178420 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:37.178362 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05d824e_3be1_46f6_8df0_2ade9675e534.slice/crio-a1499c4e086bed3b3eef48481625653b766960869ddbb32025d24cb8b8d8c45d WatchSource:0}: Error finding container a1499c4e086bed3b3eef48481625653b766960869ddbb32025d24cb8b8d8c45d: Status 404 returned error can't find the container with id a1499c4e086bed3b3eef48481625653b766960869ddbb32025d24cb8b8d8c45d Oct 31 12:31:37.406093 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.406048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:37.406272 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.406228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:37.406328 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.406270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:37.406328 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.406298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:37.406499 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.406478 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 12:31:37.406571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.406547 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls podName:746d043e-1f60-464f-ba50-6a437eb9089f nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.406529791 +0000 UTC m=+38.203967841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sckc2" (UID: "746d043e-1f60-464f-ba50-6a437eb9089f") : secret "samples-operator-tls" not found Oct 31 12:31:37.407019 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.407000 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.406983823 +0000 UTC m=+38.204421887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:37.407103 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.407085 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:37.407159 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.407127 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.407113615 +0000 UTC m=+38.204551659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:37.407216 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.407181 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:37.407275 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.407214 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.407203807 +0000 UTC m=+38.204641854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:37.508018 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.507982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.508042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.508092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.508150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508151 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508229 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508234 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508292 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.508272229 +0000 UTC m=+38.305710292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508311 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls podName:8c751f98-9133-4738-9052-8dba6020cda6 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.50830075 +0000 UTC m=+38.305738800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls") pod "dns-default-p4psl" (UID: "8c751f98-9133-4738-9052-8dba6020cda6") : secret "dns-default-metrics-tls" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508321 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508352 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.508339414 +0000 UTC m=+38.305777463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508378 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508386 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:37.508571 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:37.508409 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.5084019 +0000 UTC m=+38.305839944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:37.969009 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:37.968972 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gjvxs" event={"ID":"f05d824e-3be1-46f6-8df0-2ade9675e534","Type":"ContainerStarted","Data":"a1499c4e086bed3b3eef48481625653b766960869ddbb32025d24cb8b8d8c45d"} Oct 31 12:31:38.973535 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:38.973496 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" event={"ID":"55bcf0cc-7eef-4cbe-9d72-140932580c56","Type":"ContainerStarted","Data":"ca27f2953fe244c854deb997a36624a4104c309f305f79f145f614f242e1bfb0"} Oct 31 12:31:38.997682 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:38.997188 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" podStartSLOduration=341.790318442 podStartE2EDuration="5m45.997172263s" totalImagesPullingTime="4.206853821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:38.995736477 +0000 UTC m=+35.793174547" watchObservedRunningTime="2026-10-31 12:31:38.997172263 +0000 UTC m=+35.794610329" Oct 31 12:31:39.505922 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.504540 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jc7gp"] Oct 31 12:31:39.524396 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.523908 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7vhj5-f2fsm"] Oct 31 12:31:39.524396 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.524026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.528364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.528207 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:31:39.528364 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.528323 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52dg7\"" Oct 31 12:31:39.533497 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.532721 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:31:39.543455 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.542941 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jc7gp"] Oct 31 12:31:39.543455 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.542985 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7vhj5-f2fsm"] Oct 31 12:31:39.543455 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.543077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4h8n\" (UniqueName: \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e60a880f-7773-41dd-9995-3b1719fe8878-crio-socket\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.632956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e60a880f-7773-41dd-9995-3b1719fe8878-data-volume\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.633256 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.633016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzbjm\" (UniqueName: \"kubernetes.io/projected/e60a880f-7773-41dd-9995-3b1719fe8878-kube-api-access-fzbjm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4h8n\" (UniqueName: \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e60a880f-7773-41dd-9995-3b1719fe8878-crio-socket\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e60a880f-7773-41dd-9995-3b1719fe8878-data-volume\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbjm\" (UniqueName: \"kubernetes.io/projected/e60a880f-7773-41dd-9995-3b1719fe8878-kube-api-access-fzbjm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.734934 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.735682 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:39.735785 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:39.735840 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:40.235819078 +0000 UTC m=+37.033257122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.736131 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e60a880f-7773-41dd-9995-3b1719fe8878-crio-socket\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.736506 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.736757 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e60a880f-7773-41dd-9995-3b1719fe8878-data-volume\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.737189 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:39.737000 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:31:39.738104 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:39.737048 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:40.237033028 +0000 UTC m=+37.034471075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : configmap "kube-rbac-proxy" not found Oct 31 12:31:39.754851 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.754800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4h8n\" (UniqueName: \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.765472 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.763489 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbjm\" (UniqueName: \"kubernetes.io/projected/e60a880f-7773-41dd-9995-3b1719fe8878-kube-api-access-fzbjm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:39.915788 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.915740 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:39.994439 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:39.994334 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" event={"ID":"fee2a93c-fe10-47e5-840c-8d8be03b2715","Type":"ContainerStarted","Data":"061fed8e866a4a43db916f9cede7be91c5c27c3261b69c8142b9121ebd46ba07"} Oct 31 12:31:40.009813 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.009747 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/0.log" Oct 31 12:31:40.009813 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.009797 2580 generic.go:356] "Generic (PLEG): container finished" podID="a8053559-57b0-468c-a369-893764b0ed4d" containerID="959edc5a50363d22554f7fd7834601befe65d51b58814a3bc639a0e16fe974b9" exitCode=255 Oct 31 12:31:40.010476 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.010456 2580 scope.go:122] "RemoveContainer" containerID="959edc5a50363d22554f7fd7834601befe65d51b58814a3bc639a0e16fe974b9" Oct 31 12:31:40.010802 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.010738 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" event={"ID":"a8053559-57b0-468c-a369-893764b0ed4d","Type":"ContainerDied","Data":"959edc5a50363d22554f7fd7834601befe65d51b58814a3bc639a0e16fe974b9"} Oct 31 12:31:40.016932 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.015325 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" podStartSLOduration=339.733371484 podStartE2EDuration="5m45.015310628s" totalImagesPullingTime="5.281939144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:40.014404677 +0000 UTC m=+36.811842755" watchObservedRunningTime="2026-10-31 12:31:40.015310628 +0000 UTC m=+36.812748695" Oct 31 12:31:40.224422 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.224389 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7vhj5-f2fsm"] Oct 31 12:31:40.240805 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.240770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:40.240960 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.240836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:40.240960 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:40.240920 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:31:40.240960 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:40.240953 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:40.241099 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:40.240997 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.240974514 +0000 UTC m=+38.038412573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : configmap "kube-rbac-proxy" not found Oct 31 12:31:40.241099 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:40.241018 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:41.241009587 +0000 UTC m=+38.038447631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:40.296966 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:40.296926 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d3094e_4769_4135_9878_d0e72089389e.slice/crio-9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29 WatchSource:0}: Error finding container 9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29: Status 404 returned error can't find the container with id 9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29 Oct 31 12:31:40.832108 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.832061 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld"] Oct 31 12:31:40.865335 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.865218 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf"] Oct 31 12:31:40.865335 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.865326 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" Oct 31 12:31:40.878389 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.878361 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 12:31:40.878715 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.878690 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ttjjk\"" Oct 31 12:31:40.878816 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.878794 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:40.884579 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.884542 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld"] Oct 31 12:31:40.884730 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.884593 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf"] Oct 31 12:31:40.884730 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.884710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" Oct 31 12:31:40.948490 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:40.948367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8rb\" (UniqueName: \"kubernetes.io/projected/418e7a93-3e5c-41d8-a0be-bf74b3919943-kube-api-access-9l8rb\") pod \"migrator-66df5d496b-6xtld\" (UID: \"418e7a93-3e5c-41d8-a0be-bf74b3919943\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" Oct 31 12:31:41.017673 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.017464 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" event={"ID":"236577a8-0720-452b-83fa-986cb9aeba14","Type":"ContainerStarted","Data":"7942d4111a4ce9f2a98b6a9f1500821bc4bec5822e47bf43d1c509420eacbc28"} Oct 31 12:31:41.020404 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.020370 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" event={"ID":"55d3094e-4769-4135-9878-d0e72089389e","Type":"ContainerStarted","Data":"e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1"} Oct 31 12:31:41.020558 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.020431 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" event={"ID":"55d3094e-4769-4135-9878-d0e72089389e","Type":"ContainerStarted","Data":"9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29"} Oct 31 12:31:41.023918 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.023889 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/1.log" Oct 31 12:31:41.025362 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.024418 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/0.log" Oct 31 12:31:41.025362 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.024468 2580 generic.go:356] "Generic (PLEG): container finished" podID="a8053559-57b0-468c-a369-893764b0ed4d" containerID="67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692" exitCode=255 Oct 31 12:31:41.025362 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.024569 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" event={"ID":"a8053559-57b0-468c-a369-893764b0ed4d","Type":"ContainerDied","Data":"67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692"} Oct 31 12:31:41.025362 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.024616 2580 scope.go:122] "RemoveContainer" containerID="959edc5a50363d22554f7fd7834601befe65d51b58814a3bc639a0e16fe974b9" Oct 31 12:31:41.025362 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.024917 2580 scope.go:122] "RemoveContainer" containerID="67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692" Oct 31 12:31:41.025362 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.025120 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vnxg_openshift-console-operator(a8053559-57b0-468c-a369-893764b0ed4d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" podUID="a8053559-57b0-468c-a369-893764b0ed4d" Oct 31 12:31:41.049497 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.049112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8rb\" (UniqueName: \"kubernetes.io/projected/418e7a93-3e5c-41d8-a0be-bf74b3919943-kube-api-access-9l8rb\") pod \"migrator-66df5d496b-6xtld\" (UID: \"418e7a93-3e5c-41d8-a0be-bf74b3919943\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" Oct 31 12:31:41.049497 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.049190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdgb7\" (UniqueName: \"kubernetes.io/projected/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9-kube-api-access-fdgb7\") pod \"migrator-66df5d496b-h4gtf\" (UID: \"ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" Oct 31 12:31:41.065187 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.065128 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" podStartSLOduration=342.859890387 podStartE2EDuration="5m49.065108367s" totalImagesPullingTime="6.20521798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:41.043760171 +0000 UTC m=+37.841198236" watchObservedRunningTime="2026-10-31 12:31:41.065108367 +0000 UTC m=+37.862546434" Oct 31 12:31:41.071591 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.071560 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8rb\" (UniqueName: \"kubernetes.io/projected/418e7a93-3e5c-41d8-a0be-bf74b3919943-kube-api-access-9l8rb\") pod \"migrator-66df5d496b-6xtld\" (UID: \"418e7a93-3e5c-41d8-a0be-bf74b3919943\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" Oct 31 12:31:41.092804 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.092757 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" podStartSLOduration=2.092742231 podStartE2EDuration="2.092742231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:41.066383887 +0000 UTC m=+37.863821956" watchObservedRunningTime="2026-10-31 12:31:41.092742231 +0000 UTC m=+37.890180298" Oct 31 12:31:41.150148 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.149924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdgb7\" (UniqueName: \"kubernetes.io/projected/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9-kube-api-access-fdgb7\") pod \"migrator-66df5d496b-h4gtf\" (UID: \"ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" Oct 31 12:31:41.165245 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.165214 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdgb7\" (UniqueName: \"kubernetes.io/projected/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9-kube-api-access-fdgb7\") pod \"migrator-66df5d496b-h4gtf\" (UID: \"ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" Oct 31 12:31:41.194890 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.194849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" Oct 31 12:31:41.214600 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.214537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" Oct 31 12:31:41.251228 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.251197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:41.251967 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.251791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:41.251967 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.251902 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:31:41.251967 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.251917 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:41.252200 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.251973 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:43.251953511 +0000 UTC m=+40.049391557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:41.252200 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.251992 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:43.25198391 +0000 UTC m=+40.049421954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : configmap "kube-rbac-proxy" not found Oct 31 12:31:41.454418 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.454338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:41.454566 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.454516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:41.454566 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.454549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:41.454662 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.454573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:41.454789 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.454772 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 12:31:41.454835 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.454829 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls podName:746d043e-1f60-464f-ba50-6a437eb9089f nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.454811721 +0000 UTC m=+46.252249768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sckc2" (UID: "746d043e-1f60-464f-ba50-6a437eb9089f") : secret "samples-operator-tls" not found Oct 31 12:31:41.454917 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.454890 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:41.454917 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.454901 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.454885194 +0000 UTC m=+46.252323241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:41.455047 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.454939 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.454924077 +0000 UTC m=+46.252362134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:41.455110 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.455041 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:41.455159 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.455112 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.455092595 +0000 UTC m=+46.252530658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:41.548060 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:41.547775 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418e7a93_3e5c_41d8_a0be_bf74b3919943.slice/crio-4b047649a79d2cbe92d24b545df5c76874fd207025c477998458d0547c3a4164 WatchSource:0}: Error finding container 4b047649a79d2cbe92d24b545df5c76874fd207025c477998458d0547c3a4164: Status 404 returned error can't find the container with id 4b047649a79d2cbe92d24b545df5c76874fd207025c477998458d0547c3a4164 Oct 31 12:31:41.550186 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.550159 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld"] Oct 31 12:31:41.555879 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.555856 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf"] Oct 31 12:31:41.556026 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.555992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:41.556131 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.556113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:41.556200 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556130 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:41.556250 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556209 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.556178871 +0000 UTC m=+46.353616915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:41.556315 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556286 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:41.556315 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556309 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:41.556410 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.556326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:41.556410 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556353 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.556338814 +0000 UTC m=+46.353776864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:41.556509 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556410 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:41.556509 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.556430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:41.556509 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556440 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:41.556509 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556479 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.556466346 +0000 UTC m=+46.353904391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:41.556658 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556525 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 12:31:41.556658 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:41.556564 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls podName:8c751f98-9133-4738-9052-8dba6020cda6 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:49.55655345 +0000 UTC m=+46.353991499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls") pod "dns-default-p4psl" (UID: "8c751f98-9133-4738-9052-8dba6020cda6") : secret "dns-default-metrics-tls" not found Oct 31 12:31:41.558022 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:41.557997 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5b413d_7afc_4d3d_ae8f_12c9ce8439e9.slice/crio-ba969bba97dc07b3af04fa25d4678cbd2f552fc0afa21e7dcebfa4e970a4468c WatchSource:0}: Error finding container ba969bba97dc07b3af04fa25d4678cbd2f552fc0afa21e7dcebfa4e970a4468c: Status 404 returned error can't find the container with id ba969bba97dc07b3af04fa25d4678cbd2f552fc0afa21e7dcebfa4e970a4468c Oct 31 12:31:41.732599 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.732566 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sjz88"] Oct 31 12:31:41.750832 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.750797 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sjz88"] Oct 31 12:31:41.750985 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.750825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.755181 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.755153 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 12:31:41.755457 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.755414 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9rdmw\"" Oct 31 12:31:41.755604 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.755590 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 12:31:41.755747 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.755732 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 12:31:41.755809 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.755783 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 12:31:41.858976 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.858938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7e7e895-264d-4057-a3cc-602abc783f04-signing-key\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.859134 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.858989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7e7e895-264d-4057-a3cc-602abc783f04-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.859183 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.859138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd54w\" (UniqueName: \"kubernetes.io/projected/d7e7e895-264d-4057-a3cc-602abc783f04-kube-api-access-qd54w\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.960036 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.959998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd54w\" (UniqueName: \"kubernetes.io/projected/d7e7e895-264d-4057-a3cc-602abc783f04-kube-api-access-qd54w\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.960159 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.960141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7e7e895-264d-4057-a3cc-602abc783f04-signing-key\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.960217 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.960198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7e7e895-264d-4057-a3cc-602abc783f04-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.960842 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.960817 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7e7e895-264d-4057-a3cc-602abc783f04-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.962932 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.962912 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7e7e895-264d-4057-a3cc-602abc783f04-signing-key\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.972746 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.972724 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd54w\" (UniqueName: \"kubernetes.io/projected/d7e7e895-264d-4057-a3cc-602abc783f04-kube-api-access-qd54w\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:41.984208 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:41.984158 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9v4bh_a069e898-8264-4573-b885-2528dadb0305/dns-node-resolver/0.log" Oct 31 12:31:42.030679 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:42.030648 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" event={"ID":"ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9","Type":"ContainerStarted","Data":"ba969bba97dc07b3af04fa25d4678cbd2f552fc0afa21e7dcebfa4e970a4468c"} Oct 31 12:31:42.032358 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:42.032339 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/1.log" Oct 31 12:31:42.032734 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:42.032712 2580 scope.go:122] "RemoveContainer" containerID="67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692" Oct 31 12:31:42.032968 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:42.032946 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vnxg_openshift-console-operator(a8053559-57b0-468c-a369-893764b0ed4d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" podUID="a8053559-57b0-468c-a369-893764b0ed4d" Oct 31 12:31:42.033853 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:42.033828 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" event={"ID":"418e7a93-3e5c-41d8-a0be-bf74b3919943","Type":"ContainerStarted","Data":"4b047649a79d2cbe92d24b545df5c76874fd207025c477998458d0547c3a4164"} Oct 31 12:31:42.076551 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:42.076526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:42.517211 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:42.517172 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sjz88"] Oct 31 12:31:42.523010 ip-10-0-135-41 kubenswrapper[2580]: W1031 12:31:42.522975 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e7e895_264d_4057_a3cc_602abc783f04.slice/crio-d0835394a69db459aff40fc07bd116c983fd805b51d9ca5101647427aeb2cb7e WatchSource:0}: Error finding container d0835394a69db459aff40fc07bd116c983fd805b51d9ca5101647427aeb2cb7e: Status 404 returned error can't find the container with id d0835394a69db459aff40fc07bd116c983fd805b51d9ca5101647427aeb2cb7e Oct 31 12:31:42.784962 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:42.784814 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qkx64_0bcc043c-d8ac-4f2a-ae8d-3e233db2842d/node-ca/0.log" Oct 31 12:31:43.044363 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.043835 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" event={"ID":"bf409f2b-9f95-4f84-bbd6-d9cc841fc074","Type":"ContainerStarted","Data":"ebd18a1c269a7509f2180226f2debb2bb2d5d9c33c26f796e770745167e26e1b"} Oct 31 12:31:43.046192 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.045882 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gjvxs" event={"ID":"f05d824e-3be1-46f6-8df0-2ade9675e534","Type":"ContainerStarted","Data":"350e135e1daf171634ea56173726f09c3629f3b43726bec505533fff4a5903e5"} Oct 31 12:31:43.046192 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.046029 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:43.047688 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.047621 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bqcvc" event={"ID":"e040a356-8196-40c6-8f2c-a94bb4c6bfe4","Type":"ContainerStarted","Data":"74d8299d5f62768baf1755ba5b10f3a3c2eb8f42aba30bed32c5818d4f7e3181"} Oct 31 12:31:43.050170 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.049799 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" event={"ID":"d7e7e895-264d-4057-a3cc-602abc783f04","Type":"ContainerStarted","Data":"d64f7cebcb02a3d97c3618dce331b8c33bf8d6933821fc701f96b0d99ef94fbf"} Oct 31 12:31:43.050170 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.049835 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" event={"ID":"d7e7e895-264d-4057-a3cc-602abc783f04","Type":"ContainerStarted","Data":"d0835394a69db459aff40fc07bd116c983fd805b51d9ca5101647427aeb2cb7e"} Oct 31 12:31:43.074191 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.074022 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" podStartSLOduration=340.597725849 podStartE2EDuration="5m48.074005123s" totalImagesPullingTime="7.476279274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:43.071607874 +0000 UTC m=+39.869045953" watchObservedRunningTime="2026-10-31 12:31:43.074005123 +0000 UTC m=+39.871443192" Oct 31 12:31:43.131724 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.131663 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gjvxs" podStartSLOduration=34.34824011 podStartE2EDuration="39.131643844s" totalImagesPullingTime="4.783403734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:43.104435719 +0000 UTC m=+39.901873804" watchObservedRunningTime="2026-10-31 12:31:43.131643844 +0000 UTC m=+39.929081902" Oct 31 12:31:43.169380 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.169321 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bqcvc" podStartSLOduration=2.519796492 podStartE2EDuration="10.169301206s" totalImagesPullingTime="7.649504714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:43.168952806 +0000 UTC m=+39.966390883" watchObservedRunningTime="2026-10-31 12:31:43.169301206 +0000 UTC m=+39.966739271" Oct 31 12:31:43.169552 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.169429 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" podStartSLOduration=2.169423769 podStartE2EDuration="2.169423769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:43.132167761 +0000 UTC m=+39.929605829" watchObservedRunningTime="2026-10-31 12:31:43.169423769 +0000 UTC m=+39.966861837" Oct 31 12:31:43.275508 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.275459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:43.275683 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.275522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:43.275753 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:43.275701 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:43.275812 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:43.275760 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.275745854 +0000 UTC m=+44.073183911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:43.275812 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:43.275808 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 12:31:43.275911 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:43.275840 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.275828887 +0000 UTC m=+44.073266945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : configmap "kube-rbac-proxy" not found Oct 31 12:31:43.384215 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.384139 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqcvc_e040a356-8196-40c6-8f2c-a94bb4c6bfe4/serve-healthcheck-canary/0.log" Oct 31 12:31:43.993467 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.993430 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:43.993642 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.993495 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:43.993892 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:43.993878 2580 scope.go:122] "RemoveContainer" containerID="67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692" Oct 31 12:31:43.994098 ip-10-0-135-41 kubenswrapper[2580]: E1031 12:31:43.994075 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vnxg_openshift-console-operator(a8053559-57b0-468c-a369-893764b0ed4d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" podUID="a8053559-57b0-468c-a369-893764b0ed4d" Oct 31 12:31:44.057247 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:44.056754 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" event={"ID":"ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9","Type":"ContainerStarted","Data":"8a3ba16553607910e018c2fb08c238e589d57bf5cd5a64f142d11b590f083fbf"} Oct 31 12:31:44.059037 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:44.058755 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" event={"ID":"418e7a93-3e5c-41d8-a0be-bf74b3919943","Type":"ContainerStarted","Data":"73b5cd21f9606dc6258df1e96cfaff855d0c10894f06c729ab7b9f024aee43cc"} Oct 31 12:31:44.077548 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:44.077488 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" podStartSLOduration=1.936804363 podStartE2EDuration="4.077475544s" totalImagesPullingTime="2.140671181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:44.075941222 +0000 UTC m=+40.873379289" watchObservedRunningTime="2026-10-31 12:31:44.077475544 +0000 UTC m=+40.874913609" Oct 31 12:31:44.097833 ip-10-0-135-41 kubenswrapper[2580]: I1031 12:31:44.097778 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" podStartSLOduration=1.806248909 podStartE2EDuration="4.097761588s" totalImagesPullingTime="2.291512679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:44.096292955 +0000 UTC m=+40.893731022" watchObservedRunningTime="2026-10-31 12:31:44.097761588 +0000 UTC m=+40.895199655" Oct 31 12:31:45.761745 ip-10-0-135-41 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 12:31:45.774012 ip-10-0-135-41 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 12:31:45.774174 ip-10-0-135-41 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 12:31:45.774223 ip-10-0-135-41 systemd[1]: kubelet.service: Consumed 3.204s CPU time. Oct 31 12:31:45.792347 ip-10-0-135-41 systemd[1]: Starting Kubernetes Kubelet... Oct 31 12:31:45.850478 ip-10-0-135-41 kubenswrapper[5074]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:45.850478 ip-10-0-135-41 kubenswrapper[5074]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 12:31:45.850478 ip-10-0-135-41 kubenswrapper[5074]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:45.850886 ip-10-0-135-41 kubenswrapper[5074]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 12:31:45.852583 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852510 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.852583 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852583 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852588 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852591 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852594 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852598 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852601 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852603 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852609 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852612 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852615 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852618 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852620 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852623 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852626 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852646 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852651 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852655 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852658 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852661 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.852673 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852663 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852666 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852669 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852671 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852674 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852677 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852680 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852683 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852686 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852689 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852691 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852694 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852697 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852699 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852702 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852704 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852708 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852711 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852714 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852717 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.853146 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852719 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852721 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852724 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852727 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852729 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852732 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852734 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852737 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852742 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852747 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852750 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852753 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852756 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852759 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852762 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852764 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852767 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852770 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852773 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.853626 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852776 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852778 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852781 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852783 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852786 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852789 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852792 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852797 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852801 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852805 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852808 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852811 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852814 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852816 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852819 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852822 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852825 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852827 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852830 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.854094 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852833 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852836 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852839 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852842 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852844 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852847 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852851 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852855 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852859 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852866 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852870 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852874 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852878 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852882 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852886 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852890 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852894 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852898 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852902 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852906 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.854551 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852910 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852915 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852919 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852923 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852927 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852932 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852936 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.852941 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853727 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853735 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853737 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853741 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853744 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853746 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853749 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853752 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853754 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853757 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853760 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853763 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.855053 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853766 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853768 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853771 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853774 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853777 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853780 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853782 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853785 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853788 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853790 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853793 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853795 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853798 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853802 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853804 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853807 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853810 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853813 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853817 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.855529 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853819 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853822 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853824 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853827 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853830 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853832 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853835 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853838 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853840 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853843 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853846 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853849 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853854 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853856 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853859 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853862 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853864 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853867 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853870 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.855991 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853872 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853875 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853877 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853880 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853882 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853885 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853889 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853893 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853897 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853900 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853903 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853906 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853916 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853919 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853921 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853924 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853927 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853929 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853932 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853935 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.856479 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853937 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853940 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853942 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853945 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853947 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853950 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853953 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853955 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853958 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853960 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853963 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853965 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853968 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853971 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853973 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853976 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853978 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853981 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853984 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853986 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.856981 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853989 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853992 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.853997 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854000 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854003 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854005 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854008 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854010 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854013 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854016 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854018 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854022 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854025 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854028 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854030 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854033 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854486 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854491 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854494 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.857495 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854497 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854500 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854503 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854505 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854508 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854511 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854515 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854519 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854522 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854525 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854528 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854531 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854534 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854536 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854539 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854545 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854548 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854551 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854554 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854557 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.857975 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854563 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854566 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854569 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854572 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854574 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854577 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854580 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854583 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854586 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854589 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854591 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854594 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854597 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854599 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854602 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854604 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854607 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854610 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854612 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854614 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.858480 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854617 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854619 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854623 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854625 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854643 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854647 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854649 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854655 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854658 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854661 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854664 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854666 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854669 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854671 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854674 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854676 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854679 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854681 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854684 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854686 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.859216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854689 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854692 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854694 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854697 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854699 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854702 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854705 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854708 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854710 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854713 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854715 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854718 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854721 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854723 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854726 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854729 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854731 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854734 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854736 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.860024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854740 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854743 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854746 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854748 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854751 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854754 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854756 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854759 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854763 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854766 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854769 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854772 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854774 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854777 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854779 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854782 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854784 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854787 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854789 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.860550 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854792 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854795 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854798 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854800 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.854803 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854897 5074 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854916 5074 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854924 5074 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854929 5074 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854934 5074 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854937 5074 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854941 5074 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854950 5074 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854955 5074 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854958 5074 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854962 5074 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854965 5074 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854968 5074 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854971 5074 flags.go:64] FLAG: --cgroup-root="" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854974 5074 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854977 5074 flags.go:64] FLAG: --client-ca-file="" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854980 5074 flags.go:64] FLAG: --cloud-provider="external" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854983 5074 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854987 5074 flags.go:64] FLAG: --cluster-domain="" Oct 31 12:31:45.861050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854990 5074 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854993 5074 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.854996 5074 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855000 5074 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855004 5074 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855007 5074 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855010 5074 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855013 5074 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855016 5074 flags.go:64] FLAG: --contention-profiling="false" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855021 5074 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855024 5074 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855027 5074 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855030 5074 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855035 5074 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855038 5074 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855041 5074 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855044 5074 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855047 5074 flags.go:64] FLAG: --enable-server="true" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855051 5074 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855055 5074 flags.go:64] FLAG: --event-burst="100" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855058 5074 flags.go:64] FLAG: --event-qps="50" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855062 5074 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855065 5074 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855069 5074 flags.go:64] FLAG: --eviction-hard="" Oct 31 12:31:45.861616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855074 5074 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855077 5074 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855080 5074 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855083 5074 flags.go:64] FLAG: --eviction-soft="" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855086 5074 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855089 5074 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855092 5074 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855095 5074 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855098 5074 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855101 5074 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855105 5074 flags.go:64] FLAG: --feature-gates="" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855109 5074 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855112 5074 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855115 5074 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855118 5074 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855121 5074 flags.go:64] FLAG: --healthz-port="10248" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855125 5074 flags.go:64] FLAG: --help="false" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855129 5074 flags.go:64] FLAG: --hostname-override="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855133 5074 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855136 5074 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855140 5074 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855143 5074 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855147 5074 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 12:31:45.862227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855150 5074 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855153 5074 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855156 5074 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855159 5074 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855162 5074 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855166 5074 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855168 5074 flags.go:64] FLAG: --kube-reserved="" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855171 5074 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855174 5074 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855178 5074 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855181 5074 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855184 5074 flags.go:64] FLAG: --lock-file="" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855188 5074 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855191 5074 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855194 5074 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855199 5074 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855202 5074 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855205 5074 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855208 5074 flags.go:64] FLAG: --logging-format="text" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855211 5074 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855215 5074 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855218 5074 flags.go:64] FLAG: --manifest-url="" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855221 5074 flags.go:64] FLAG: --manifest-url-header="" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855225 5074 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855228 5074 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 12:31:45.862797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855232 5074 flags.go:64] FLAG: --max-pods="110" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855237 5074 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855240 5074 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855243 5074 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855246 5074 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855249 5074 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855252 5074 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855255 5074 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855262 5074 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855266 5074 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855269 5074 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855272 5074 flags.go:64] FLAG: --pod-cidr="" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855275 5074 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855278 5074 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855281 5074 flags.go:64] FLAG: --pods-per-core="0" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855284 5074 flags.go:64] FLAG: --port="10250" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855287 5074 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855290 5074 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b7e1436149d8ecbb" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855297 5074 flags.go:64] FLAG: --qos-reserved="" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855301 5074 flags.go:64] FLAG: --read-only-port="10255" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855304 5074 flags.go:64] FLAG: --register-node="true" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855307 5074 flags.go:64] FLAG: --register-with-taints="" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855315 5074 flags.go:64] FLAG: --registry-burst="10" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855318 5074 flags.go:64] FLAG: --registry-qps="5" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855321 5074 flags.go:64] FLAG: --reserved-cpus="" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855324 5074 flags.go:64] FLAG: --reserved-memory="" Oct 31 12:31:45.863418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855328 5074 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855331 5074 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855334 5074 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855337 5074 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855340 5074 flags.go:64] FLAG: --runonce="false" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855343 5074 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855346 5074 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855351 5074 flags.go:64] FLAG: --seccomp-default="false" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855354 5074 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855357 5074 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855360 5074 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855364 5074 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855367 5074 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855369 5074 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855373 5074 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855375 5074 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855378 5074 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855381 5074 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855384 5074 flags.go:64] FLAG: --system-cgroups="" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855388 5074 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855393 5074 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855396 5074 flags.go:64] FLAG: --tls-cert-file="" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855399 5074 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855403 5074 flags.go:64] FLAG: --tls-min-version="" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855406 5074 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 12:31:45.864061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855409 5074 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855412 5074 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855415 5074 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855419 5074 flags.go:64] FLAG: --v="2" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855423 5074 flags.go:64] FLAG: --version="false" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855427 5074 flags.go:64] FLAG: --vmodule="" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855431 5074 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855435 5074 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855556 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855560 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855564 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855567 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855569 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855572 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855576 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855579 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855581 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855585 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855587 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855590 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855592 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.864775 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855595 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855597 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855600 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855603 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855605 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855608 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855611 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855613 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855616 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855619 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855621 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855624 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855627 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855644 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855648 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855650 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855653 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855655 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855658 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.865286 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855662 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855667 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855672 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855675 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855678 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855680 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855685 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855687 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855690 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855693 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855696 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855698 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855701 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855704 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855706 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855709 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855711 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855714 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855717 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855720 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.865771 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855722 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855725 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855728 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855730 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855733 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855736 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855742 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855745 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855748 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855751 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855755 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855759 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855762 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855765 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855770 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855772 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855775 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855778 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855782 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.866353 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855785 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855787 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855790 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855793 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855795 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855798 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855800 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855803 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855806 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855808 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855811 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855814 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855816 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855819 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855821 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855824 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855826 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855829 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855831 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855834 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.866945 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855837 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855840 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855843 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855845 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855848 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855850 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855853 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855856 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855859 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855862 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855864 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855869 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855872 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855875 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855877 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.855904 5074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855940 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.867481 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855944 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855947 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855950 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855952 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855955 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855958 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855961 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855963 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855966 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855969 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855971 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855974 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855976 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855979 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855982 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855984 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855987 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855990 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855993 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.868179 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855996 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.855999 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856001 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856005 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856008 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856010 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856013 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856016 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856019 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856024 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856027 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856030 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856033 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856035 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856038 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856040 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856043 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856046 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856048 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.869032 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856051 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856054 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856056 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856059 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856062 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856064 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856067 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856069 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856072 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856074 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856077 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856081 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856084 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856087 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856090 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856092 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856096 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856099 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856101 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.869627 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856104 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856108 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856112 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856115 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856118 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856120 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856123 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856126 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856128 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856131 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856134 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856136 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856139 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856142 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856144 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856147 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856149 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856152 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856155 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856157 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.870165 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856160 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856163 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856165 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856168 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856170 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856173 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856176 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856178 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856181 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856184 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856187 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856191 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856194 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856196 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856200 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856203 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856205 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856208 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856211 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856213 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.870724 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856216 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856219 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856221 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856224 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856226 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856229 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856232 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.856234 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.861726 5074 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.861746 5074 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.861769 5074 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.861774 5074 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861835 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861842 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861845 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861848 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861851 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861854 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861857 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.871206 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861859 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861862 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861865 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861868 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861870 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861873 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861876 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861879 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861882 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861885 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861888 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861891 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861893 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861896 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861899 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861902 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861904 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861914 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861918 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861921 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.871792 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861924 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861927 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861930 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861933 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861935 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861938 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861941 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861943 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861946 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861948 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861951 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861953 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861956 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861959 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861961 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861964 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861966 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861969 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861972 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.872272 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861975 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861978 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861980 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861983 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861985 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861989 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861992 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861994 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.861997 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862000 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862002 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862005 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862008 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862010 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862012 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862015 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862018 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862021 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862024 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862026 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.872734 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862029 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862031 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862034 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862036 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862039 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862042 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862044 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862047 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862049 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862052 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862055 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862057 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862060 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862063 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862065 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862069 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862074 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862077 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862080 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.873217 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862083 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862086 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862089 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862091 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862094 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862097 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862100 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862102 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862105 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862108 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862110 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862113 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862116 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862118 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862121 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862123 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862126 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862129 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862131 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862133 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.873682 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862136 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.862164 5074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862317 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862323 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862327 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862331 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862334 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862338 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862341 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862343 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862346 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862349 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862352 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862354 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862357 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862360 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.874168 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862363 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862365 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862368 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862372 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862375 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862377 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862380 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862382 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862385 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862388 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862390 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862393 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862396 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862398 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862401 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862403 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862407 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862410 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862413 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.874542 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862416 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862418 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862421 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862424 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862426 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862429 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862432 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862434 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862437 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862440 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862442 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862445 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862447 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862450 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862452 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862455 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862458 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862460 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862463 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.875311 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862465 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862467 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862470 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862473 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862475 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862478 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862480 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862483 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862485 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862488 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862490 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862492 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862495 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862498 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862500 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862502 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862505 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862508 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862510 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.876095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862514 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862517 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862520 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862522 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862525 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862528 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862530 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862533 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862535 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862538 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862541 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862543 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862546 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862549 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862551 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862554 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862556 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862559 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862561 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862564 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.876548 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862566 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862569 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862571 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862574 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862577 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862579 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862582 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862584 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862587 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862589 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862592 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862595 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862598 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862601 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862604 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.862627 5074 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 12:31:45.877127 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862678 5074 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862684 5074 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862686 5074 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862689 5074 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862692 5074 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862695 5074 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862698 5074 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862700 5074 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862703 5074 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862705 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862708 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862711 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862713 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862716 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862719 5074 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862721 5074 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862724 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862726 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862729 5074 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862731 5074 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 12:31:45.877537 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862734 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862737 5074 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862739 5074 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862742 5074 feature_gate.go:386] unrecognized feature gate: Example Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862745 5074 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862748 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862750 5074 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862753 5074 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862755 5074 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862758 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862762 5074 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862765 5074 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862767 5074 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862770 5074 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862772 5074 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862776 5074 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862780 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862784 5074 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862788 5074 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 12:31:45.878052 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862791 5074 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862794 5074 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862797 5074 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862800 5074 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862803 5074 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862805 5074 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862808 5074 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862812 5074 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862815 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862817 5074 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862820 5074 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862823 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862825 5074 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862828 5074 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862831 5074 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862833 5074 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862836 5074 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862838 5074 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862841 5074 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862843 5074 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 12:31:45.878509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862846 5074 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862849 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862851 5074 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862854 5074 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862858 5074 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862861 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862864 5074 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862866 5074 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862869 5074 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862872 5074 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862874 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862879 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862882 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862884 5074 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862888 5074 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862890 5074 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862893 5074 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862895 5074 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862898 5074 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 12:31:45.879036 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862900 5074 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862903 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862906 5074 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862908 5074 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862910 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862913 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862916 5074 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862918 5074 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862920 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862923 5074 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862926 5074 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862928 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862931 5074 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862933 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862936 5074 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862938 5074 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862941 5074 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862943 5074 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862946 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 12:31:45.879487 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862949 5074 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862951 5074 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862954 5074 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862957 5074 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862959 5074 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862963 5074 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862966 5074 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862970 5074 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:45.862973 5074 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.863024 5074 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.864035 5074 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.864091 5074 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.864263 5074 server.go:1014] "Starting client certificate rotation" Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.864421 5074 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.864490 5074 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:26:03 +0000 UTC" deadline="2028-05-16 09:33:20.611033914 +0000 UTC" Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.864535 5074 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13509h1m34.746501891s" Oct 31 12:31:45.880062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.864743 5074 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:45.880461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.865764 5074 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 12:31:45.880461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.867062 5074 log.go:25] "Validated CRI v1 runtime API" Oct 31 12:31:45.880461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.869233 5074 log.go:25] "Validated CRI v1 image API" Oct 31 12:31:45.880461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.869992 5074 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 12:31:45.880461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.872008 5074 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 db496fcc-61ff-49b6-8276-2d61b4f4b1e2:/dev/nvme0n1p4 f6b76be6-7865-4874-9bd0-c0ac52a690ae:/dev/nvme0n1p3] Oct 31 12:31:45.880617 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.872027 5074 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/034cd7c334403750b4defa906b0e09b93223e3e3d5b1a17ec4691b6a2cce323c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/034cd7c334403750b4defa906b0e09b93223e3e3d5b1a17ec4691b6a2cce323c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1375b84931ed41b9380d19cb4f2dfad25207356b545b746f32cfb042a9706ac3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1375b84931ed41b9380d19cb4f2dfad25207356b545b746f32cfb042a9706ac3/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34b30d9b6d3b64aa18e694b99251502e5d7fffc7510f82a6415edd2a779ceccd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34b30d9b6d3b64aa18e694b99251502e5d7fffc7510f82a6415edd2a779ceccd/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/353ac1b3869e6ffb9011f953aaf415aadf741369d91fdd9b0176bece61c6c29a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/353ac1b3869e6ffb9011f953aaf415aadf741369d91fdd9b0176bece61c6c29a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4853821b0334d14c66f20cc1ff71c57dbd939417156d20a6f09c401610e96076/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4853821b0334d14c66f20cc1ff71c57dbd939417156d20a6f09c401610e96076/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a2660e80bf82352c5db70044254f189f0032d30f8ca9d91d95c670a2bdb160e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a2660e80bf82352c5db70044254f189f0032d30f8ca9d91d95c670a2bdb160e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b047649a79d2cbe92d24b545df5c76874fd207025c477998458d0547c3a4164/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b047649a79d2cbe92d24b545df5c76874fd207025c477998458d0547c3a4164/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/571d12f96a06f1644c4d7dc77e1863e14c5570d657c52f9b2e41f8bad45d3aed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/571d12f96a06f1644c4d7dc77e1863e14c5570d657c52f9b2e41f8bad45d3aed/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74722bde183b22d5645720fd42e05bd85a6b4d8e89e24e382af767b234e855aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74722bde183b22d5645720fd42e05bd85a6b4d8e89e24e382af767b234e855aa/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cfad452ea40a72182fc93d112e4ffeb0236974801cb84dbe6bd42b96e5a8ea5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cfad452ea40a72182fc93d112e4ffeb0236974801cb84dbe6bd42b96e5a8ea5/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02/userdata/shm major:0 minor:53 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8514d431fc512977b1a1605881be8be2bb1b917cfe73bf7e8e3aca8209503a60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8514d431fc512977b1a1605881be8be2bb1b917cfe73bf7e8e3aca8209503a60/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/915733798ecb455c1a10ae60de736827ff18469f26da3a6cbe871bc903d873b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/915733798ecb455c1a10ae60de736827ff18469f26da3a6cbe871bc903d873b1/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/979f00a02414c60112e49ee9012cba264a0369fa8653a7e48489ef1c0200110c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/979f00a02414c60112e49ee9012cba264a0369fa8653a7e48489ef1c0200110c/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1499c4e086bed3b3eef48481625653b766960869ddbb32025d24cb8b8d8c45d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1499c4e086bed3b3eef48481625653b766960869ddbb32025d24cb8b8d8c45d/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a27a74f42953674491bf7a41068040af323bca7609f05ef24d11251e595822f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a27a74f42953674491bf7a41068040af323bca7609f05ef24d11251e595822f5/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a93a0b3166b3bfc67fcb0fe80b254a7438f49ff4f300cd62bd088f3168dffb64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a93a0b3166b3bfc67fcb0fe80b254a7438f49ff4f300cd62bd088f3168dffb64/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4/userdata/shm major:0 minor:54 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba969bba97dc07b3af04fa25d4678cbd2f552fc0afa21e7dcebfa4e970a4468c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba969bba97dc07b3af04fa25d4678cbd2f552fc0afa21e7dcebfa4e970a4468c/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c45f4aaf555deb61468e74093c2c65be2edcaea0313d9ac73aee2f7a7de02d60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c45f4aaf555deb61468e74093c2c65be2edcaea0313d9ac73aee2f7a7de02d60/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0835394a69db459aff40fc07bd116c983fd805b51d9ca5101647427aeb2cb7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0835394a69db459aff40fc07bd116c983fd805b51d9ca5101647427aeb2cb7e/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7883d6e70ed6e5d818f140fd7b5963430ab4d60a23a9073ede8f24eedcf7cb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7883d6e70ed6e5d818f140fd7b5963430ab4d60a23a9073ede8f24eedcf7cb2/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f33883c96c28647af1b557d2939ca7f9aaa5215fd7bb2d2875a7a45dacbe96ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f33883c96c28647af1b557d2939ca7f9aaa5215fd7bb2d2875a7a45dacbe96ba/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa6cc5e19d492cf2cb49b2106f61c27e5ff4c0d9dc01b6639dc114076e24f040/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa6cc5e19d492cf2cb49b2106f61c27e5ff4c0d9dc01b6639dc114076e24f040/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fce15112b54ab63d32596f08ee801da328b9b48d5522eaa3f1f7e7d3489ea9fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fce15112b54ab63d32596f08ee801da328b9b48d5522eaa3f1f7e7d3489ea9fc/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0554ee01-1433-4bd7-9a51-d74c757af46a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0554ee01-1433-4bd7-9a51-d74c757af46a/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d/volumes/kubernetes.io~projected/kube-api-access-n7tbp:{mountpoint:/var/lib/kubelet/pods/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d/volumes/kubernetes.io~projected/kube-api-access-n7tbp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236577a8-0720-452b-83fa-986cb9aeba14/volumes/kubernetes.io~projected/kube-api-access-mq2fm:{mountpoint:/var/lib/kubelet/pods/236577a8-0720-452b-83fa-986cb9aeba14/volumes/kubernetes.io~projected/kube-api-access-mq2fm major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236577a8-0720-452b-83fa-986cb9aeba14/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/236577a8-0720-452b-83fa-986cb9aeba14/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f23f959-82b3-48cc-8f17-917d8c8c6f6b/volumes/kubernetes.io~projected/kube-api-access-wxc57:{mountpoint:/var/lib/kubelet/pods/2f23f959-82b3-48cc-8f17-917d8c8c6f6b/volumes/kubernetes.io~projected/kube-api-access-wxc57 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~projected/kube-api-access-ntcw7:{mountpoint:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~projected/kube-api-access-ntcw7 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/418e7a93-3e5c-41d8-a0be-bf74b3919943/volumes/kubernetes.io~projected/kube-api-access-9l8rb:{mountpoint:/var/lib/kubelet/pods/418e7a93-3e5c-41d8-a0be-bf74b3919943/volumes/kubernetes.io~projected/kube-api-access-9l8rb major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volumes/kubernetes.io~projected/kube-api-access-fv66w:{mountpoint:/var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volumes/kubernetes.io~projected/kube-api-access-fv66w major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55bcf0cc-7eef-4cbe-9d72-140932580c56/volumes/kubernetes.io~projected/kube-api-access-p9r6x:{mountpoint:/var/lib/kubelet/pods/55bcf0cc-7eef-4cbe-9d72-140932580c56/volumes/kubernetes.io~projected/kube-api-access-p9r6x major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55bcf0cc-7eef-4cbe-9d72-140932580c56/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/55bcf0cc-7eef-4cbe-9d72-140932580c56/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55d3094e-4769-4135-9878-d0e72089389e/volumes/kubernetes.io~projected/kube-api-access-h4h8n:{mountpoint:/var/lib/kubelet/pods/55d3094e-4769-4135-9878-d0e72089389e/volumes/kubernetes.io~projected/kube-api-access-h4h8n major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~projected/kube-api-access-8pmcf:{mountpoint:/var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~projected/kube-api-access-8pmcf major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59030007-53c1-42c9-baec-a4d2ec90f3dd/volumes/kubernetes.io~projected/kube-api-access-757tm:{mountpoint:/var/lib/kubelet/pods/59030007-53c1-42c9-baec-a4d2ec90f3dd/volumes/kubernetes.io~projected/kube-api-access-757tm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b057b6d-b9d1-4642-95f3-41ba5498c24d/volumes/kubernetes.io~projected/kube-api-access-fgbcf:{mountpoint:/var/lib/kubelet/pods/5b057b6d-b9d1-4642-95f3-41ba5498c24d/volumes/kubernetes.io~projected/kube-api-access-fgbcf major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/719b7f16-3b6a-402e-8301-7b498a9ad131/volumes/kubernetes.io~projected/kube-api-access-bknk7:{mountpoint:/var/lib/kubelet/pods/719b7f16-3b6a-402e-8301-7b498a9ad131/volumes/kubernetes.io~projected/kube-api-access-bknk7 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/746d043e-1f60-464f-ba50-6a437eb9089f/volumes/kubernetes.io~projected/kube-api-access-6dxwp:{mountpoint:/var/lib/kubelet/pods/746d043e-1f60-464f-ba50-6a437eb9089f/volumes/kubernetes.io~projected/kube-api-access-6dxwp major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c751f98-9133-4738-9052-8dba6020cda6/volumes/kubernetes.io~projected/kube-api-access-skgq6:{mountpoint:/var/lib/kubelet/pods/8c751f98-9133-4738-9052-8dba6020cda6/volumes/kubernetes.io~projected/kube-api-access-skgq6 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cea9a4e-e914-40b6-9237-345be4aff28c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8cea9a4e-e914-40b6-9237-345be4aff28c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c7c8096-81d8-4c14-b102-08569d25da31/volumes/kubernetes.io~projected/kube-api-access-2pc8z:{mountpoint:/var/lib/kubelet/pods/9c7c8096-81d8-4c14-b102-08569d25da31/volumes/kubernetes.io~projected/kube-api-access-2pc8z major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a069e898-8264-4573-b885-2528dadb0305/volumes/kubernetes.io~projected/kube-api-access-6pqtb:{mountpoint:/var/lib/kubelet/pods/a069e898-8264-4573-b885-2528dadb0305/volumes/kubernetes.io~projected/kube-api-access-6pqtb major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~projected/kube-api-access-klpft:{mountpoint:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~projected/kube-api-access-klpft major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8053559-57b0-468c-a369-893764b0ed4d/volumes/kubernetes.io~projected/kube-api-access-khg96:{mountpoint:/var/lib/kubelet/pods/a8053559-57b0-468c-a369-893764b0ed4d/volumes/kubernetes.io~projected/kube-api-access-khg96 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8053559-57b0-468c-a369-893764b0ed4d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a8053559-57b0-468c-a369-893764b0ed4d/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf409f2b-9f95-4f84-bbd6-d9cc841fc074/volumes/kubernetes.io~projected/kube-api-access-j5nm4:{mountpoint:/var/lib/kubelet/pods/bf409f2b-9f95-4f84-bbd6-d9cc841fc074/volumes/kubernetes.io~projected/kube-api-access-j5nm4 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c765c0f2-1bc6-4eee-9fde-317749984f5c/volumes/kubernetes.io~projected/kube-api-access-w2ppx:{mountpoint:/var/lib/kubelet/pods/c765c0f2-1bc6-4eee-9fde-317749984f5c/volumes/kubernetes.io~projected/kube-api-access-w2ppx major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d409cf23-c109-4dfb-a569-5041531269dc/volumes/kubernetes.io~projected/kube-api-access-99f5v:{mountpoint:/var/lib/kubelet/pods/d409cf23-c109-4dfb-a569-5041531269dc/volumes/kubernetes.io~projected/kube-api-access-99f5v major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7e7e895-264d-4057-a3cc-602abc783f04/volumes/kubernetes.io~projected/kube-api-access-qd54w:{mountpoint:/var/lib/kubelet/pods/d7e7e895-264d-4057-a3cc-602abc783f04/volumes/kubernetes.io~projected/kube-api-access-qd54w major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7e7e895-264d-4057-a3cc-602abc783f04/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/d7e7e895-264d-4057-a3cc-602abc783f04/volumes/kubernetes.io~secret/signing-key major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dee0dda1-07c9-43fc-b7d6-f09b502d0149/volumes/kubernetes.io~projected/kube-api-access-nvv9r:{mountpoint:/var/lib/kubelet/pods/dee0dda1-07c9-43fc-b7d6-f09b502d0149/volumes/kubernetes.io~projected/kube-api-access-nvv9r major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e040a356-8196-40c6-8f2c-a94bb4c6bfe4/volumes/kubernetes.io~projected/kube-api-access-zxjsf:{mountpoint:/var/lib/kubelet/pods/e040a356-8196-40c6-8f2c-a94bb4c6bfe4/volumes/kubernetes.io~projected/kube-api-access-zxjsf major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e040a356-8196-40c6-8f2c-a94bb4c6bfe4/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e040a356-8196-40c6-8f2c-a94bb4c6bfe4/volumes/kubernetes.io~secret/cert major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e60a880f-7773-41dd-9995-3b1719fe8878/volumes/kubernetes.io~projected/kube-api-access-fzbjm:{mountpoint:/var/lib/kubelet/pods/e60a880f-7773-41dd-9995-3b1719fe8878/volumes/kubernetes.io~projected/kube-api-access-fzbjm major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d/volumes/kubernetes.io~projected/kube-api-access-s7tr4:{mountpoint:/var/lib/kubelet/pods/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d/volumes/kubernetes.io~projected/kube-api-access-s7tr4 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9/volumes/kubernetes.io~projected/kube-api-access-fdgb7:{mountpoint:/var/lib/kubelet/pods/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9/volumes/kubernetes.io~projected/kube-api-access-fdgb7 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f05d824e-3be1-46f6-8df0-2ade9675e534/volumes/kubernetes.io~projected/kube-api-access-c4sk5:{mountpoint:/var/lib/kubelet/pods/f05d824e-3be1-46f6-8df0-2ade9675e534/volumes/kubernetes.io~projected/kube-api-access-c4sk5 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~projected/kube-api-access-sxhlm:{mountpoint:/var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~projected/kube-api-access-sxhlm major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb0b76f8-eb8d-45ee-afec-b393223b0504/volumes/kubernetes.io~projected/kube-api-access-wzhdx:{mountpoint:/var/lib/kubelet/pods/fb0b76f8-eb8d-45ee-afec-b393223b0504/volumes/kubernetes.io~projected/kube-api-access-wzhdx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fee2a93c-fe10-47e5-840c-8d8be03b2715/volumes/kubernetes.io~projected/kube-api-access-zqrdr:{mountpoint:/var/lib/kubelet/pods/fee2a93c-fe10-47e5-840c-8d8be03b2715/volumes/kubernetes.io~projected/kube-api-access-zqrdr major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fee2a93c-fe10-47e5-840c-8d8be03b2715/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fee2a93c-fe10-47e5-840c-8d8be03b2715/volumes/kubernetes.io~secret/serving-cert major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4a49e904ba0d5b169f2231f9dd40d531c60b95aa0ceb5bb79a6969351d7ce7bd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/31068d7c960ae38d1e2624f20c791a6b4bfa94ae0605548437f05336dcd6e800/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e842a436a352d869affd9598f32a5cefc9d204b2b3f77ed4d57d8b9032d2f469/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/376d9587d06463ba11b45711c1748041514accc251f5bfe9336149775b8f22da/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0fa259dc50fc7d6fc40dd035a2e498a418e4226d455af26b91f31df304534cca/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ef2048560d840ff378ea11b3271dcb0aa4ef1026eb0801da26d8828d81e10555/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/44399c43b558a332aaada2517539886463a7b0235e7e8035f014fb54c54ac556/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9692e4401c400ead45920deda19f95d778f1d58bd6b3583e0f45f0af7cb5e943/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0820eea27163d2b5b031e65b901a59637288e4883b772e90ef21a2c2009ff3f4/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ef550dc1fc300086c220fad91ca84ac1c77851ef5b0476850951398379f16d54/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a34a6ce68c54fcf9e8414cb4fcd5f30bce062cab6c0e2e933ae7bce3f8531e77/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/7124604f03eee4b5464fe52d131136ca88586f04e2c7cf8aa440a41ff4f439b3/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5c518d9635f09fa364aa2e58e7993d51e2bb8e2eb02b07e271c46f029bdb009d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b59c9d47c8929e12f26a1c0fa7e780c97fdd56040bd39491af4960879b85eb0c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/08c27e5c73910499230ae3da5d8ca3ce09ed020090800071b7a1dbfb61f97516/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b5351f7588aba199c79b46edd90085f16ae4b7210e570ea7c42a4045497ed767/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/283f17d43aaa6830fbe54b9cdfcce1084431ff8bc63dc9ebed85eb0f0c0932e2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/654cbcddfdc2a7ae43043036b6101d406280635075479b9082ddaa84515e4b6c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e93cc6d59cb17772227928fe98268745b8c7ab28f90cb408d2dbb719a83cc271/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/517579526bf830dc5b51431431fb3a93aea821ebb9df2de479ba363f5287974a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1157c35889a9c59c8c4c9f0c9b94688d1e2dee481db6282fcf5f348adda0b9eb/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/55b55e4ce73217ac4550263db9ab32372adec53038b0a8736f3c101d13c9d3ff/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5950e1deed0875d5e7d42a7b456e60ca91c33cdb2a7a33212fecc23749dd5f90/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/27f3ed08927693213cb94a5539d6b8cee4a8a63c264eca763d0404a498072533/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/278a1327b24ac5a516d6720032ded57151239711f4dc7eee08bbbde562eebaee/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/e0ef7376ac1bd1249caa60b37593a5419871a665dc74692b0f41abf91f7dccad/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/a8a5b37ff1079f5225077e9795d590c405c5b821119f870655d3962caa8b5fe9/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/018094251be877363c0dd53f0a9851a297adbaf0b6edb084623f8290391f5ffd/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/5a981e93d0c90978df5fa3997a35c12e754f5be926f595123759c132587efbac/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/eb597aaedda0c3314ea4dc6312df76a2f1860113282425c14a4c0a3781b16579/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/965a74bb074062bb316e96b4d713df3abe6e719efabd8b97059036562ac60f29/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/4d234370124ea82f32a86faf83c25df76189bf2ff8da11c92b4b5775c67a35cd/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/26614ed9840a7b1e98a93c529b1305c5daa3ed2feb5946017741688c78b04a9d/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/bfbc0c24e29c6aeccc90eed28d5a2a3bfd7df37ba93cdfb84ec5ad5c3be4b952/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/4a26bd5abb4bb5772b5168fcaf9f53f8f8a2bc71f2e04ad6789dcf4d8e2ddd13/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/8584e93e9f99c5fdaacbc529201cf72473a458a42f7ba34853c66538c6be35d2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/d02b212457dc50d356c0c4f217cf7fdb9f37ddf710e2a40b5a161762cb444d9a/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/46a5d2ca7bddcfde729f1c595ccba56ae0d9cd2f819462286035de8656caedb6/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/7e2eb4df237ccd26e236f24bb38b9a6fceda8720bade12329e60ffcaaa8c29cf/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/26ad1023a526274bb94def42c133556c39446f5f49783338f40505fedb0473a8/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/9a697599233afc883d1927d03c99b663b97a9bf2ed965664d7f656bc8d300423/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/6a6fbc7bd68dcf1aced420e92772ba7cb86935e351c0dc2ef346280c26a5b965/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/a62a5fb5bdc9a9e0c03cec93e2205eef32eeed45388e316f5a5fb8a57179f818/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/6e1d4a9c9bdf08e0c2aa7c7d1595d8aa9b2296a30cd38990b318b3714501e222/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/24137d3e6c18b22af579ebb738e1c1d45b5ebe9b0f4ed5a7ed1ed4e4842fe3b4/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/fa401fc0302c2ece8815e4d86118dcbbe2b6f1198c3300901ae23993b692cc1b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/78d6a30d6fb3da835c9c087e4feb99d009f5330aa9055dc612c99b67977f89ff/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/40900f6bf416874f65ab869ed3d882167f75c66fcf4027414735c661f191a23b/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/73a39ee0f65e20f97b2ec621868ebd34145b77d3e759c813c69ebe4cefc93320/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/7d5fe309f4b1a2e9f4ace47c1e5186e57d6f82e1e1f7de77d44439124c9b2618/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/ef3746e1b183502518930a51fd8bcba1507e7048515b7b0615c5a7ef1e19192b/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/1b6a1fb9de3493e1ffbb3fce4a9499ad8bca3b5ddabc5dd806e2b94d14318c8c/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/e2246915d154584275312e108c865a1cf2a65d9ebf3063435e5c02c2ff9735a2/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-59:{mountpoint:/var/lib/containers/storage/overlay/f6f13eb14e179b495880ebb681c9a35e10d649f385a23795f1489c89343e1e6c/merged major:0 minor:59 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/af9d1ee48fef992bc689dd81d80e43f375b1f5b2cfd21b23a8d1897259545fde/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/34dcb4e1dfdf3ddbf713093fe53e3e8277e0342f912645236cbb81cc97aca856/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/84aef7b0aa05db3b5b7b712b05e3a17ed816f48b8e3f7ae53723c3da82a5ab0d/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c8c3bd07dd566cc3dafd1fcec15b43916a7e06219ba4227a41ad4c82b86237d5/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c1e2de3a1db874feb3b2a74e5de13e603c198dd5c1cf97a9d8a2889f4e2ce888/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b5637128cb69db6b361e95e48dfa9525289c284c722a18d7c981c0bc20cf85ab/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/87d1b724acaa5fe808b231169f5408fd493998a9cfdc543f30e74d242f49933d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6e20e75c05231a648b0e8a7998faba97d43ca2e7b92e7e1da99f141a18d1cd4f/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 12:31:45.881531 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.881126 5074 manager.go:217] Machine: {Timestamp:2026-10-31 12:31:45.880650965 +0000 UTC m=+0.058368940 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100606 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e6546f1cd7437e5b80b8cbe7430e SystemUUID:ec28e654-6f1c-d743-7e5b-80b8cbe7430e BootID:d47cadb3-1bb7-47c8-817b-e1f7d5a2bad0 Filesystems:[{Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cfad452ea40a72182fc93d112e4ffeb0236974801cb84dbe6bd42b96e5a8ea5/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9/volumes/kubernetes.io~projected/kube-api-access-fdgb7 DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d/volumes/kubernetes.io~projected/kube-api-access-s7tr4 DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7883d6e70ed6e5d818f140fd7b5963430ab4d60a23a9073ede8f24eedcf7cb2/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7e7e895-264d-4057-a3cc-602abc783f04/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-59 DeviceMajor:0 DeviceMinor:59 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/746d043e-1f60-464f-ba50-6a437eb9089f/volumes/kubernetes.io~projected/kube-api-access-6dxwp DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55bcf0cc-7eef-4cbe-9d72-140932580c56/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c765c0f2-1bc6-4eee-9fde-317749984f5c/volumes/kubernetes.io~projected/kube-api-access-w2ppx DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c45f4aaf555deb61468e74093c2c65be2edcaea0313d9ac73aee2f7a7de02d60/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a93a0b3166b3bfc67fcb0fe80b254a7438f49ff4f300cd62bd088f3168dffb64/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fce15112b54ab63d32596f08ee801da328b9b48d5522eaa3f1f7e7d3489ea9fc/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf409f2b-9f95-4f84-bbd6-d9cc841fc074/volumes/kubernetes.io~projected/kube-api-access-j5nm4 DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/236577a8-0720-452b-83fa-986cb9aeba14/volumes/kubernetes.io~projected/kube-api-access-mq2fm DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e60a880f-7773-41dd-9995-3b1719fe8878/volumes/kubernetes.io~projected/kube-api-access-fzbjm DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0835394a69db459aff40fc07bd116c983fd805b51d9ca5101647427aeb2cb7e/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d409cf23-c109-4dfb-a569-5041531269dc/volumes/kubernetes.io~projected/kube-api-access-99f5v DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a8a0aad-3971-455a-8fda-f40c28d8bc13/volumes/kubernetes.io~projected/kube-api-access-fv66w DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/579b4046-76c9-4d7e-b77a-9888510a22a8/volumes/kubernetes.io~projected/kube-api-access-8pmcf DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d/volumes/kubernetes.io~projected/kube-api-access-n7tbp DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b057b6d-b9d1-4642-95f3-41ba5498c24d/volumes/kubernetes.io~projected/kube-api-access-fgbcf DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a27a74f42953674491bf7a41068040af323bca7609f05ef24d11251e595822f5/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74722bde183b22d5645720fd42e05bd85a6b4d8e89e24e382af767b234e855aa/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8053559-57b0-468c-a369-893764b0ed4d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~projected/kube-api-access-klpft DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b047649a79d2cbe92d24b545df5c76874fd207025c477998458d0547c3a4164/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a069e898-8264-4573-b885-2528dadb0305/volumes/kubernetes.io~projected/kube-api-access-6pqtb DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/418e7a93-3e5c-41d8-a0be-bf74b3919943/volumes/kubernetes.io~projected/kube-api-access-9l8rb DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1375b84931ed41b9380d19cb4f2dfad25207356b545b746f32cfb042a9706ac3/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e040a356-8196-40c6-8f2c-a94bb4c6bfe4/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55bcf0cc-7eef-4cbe-9d72-140932580c56/volumes/kubernetes.io~projected/kube-api-access-p9r6x DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e040a356-8196-40c6-8f2c-a94bb4c6bfe4/volumes/kubernetes.io~projected/kube-api-access-zxjsf DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f33883c96c28647af1b557d2939ca7f9aaa5215fd7bb2d2875a7a45dacbe96ba/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a2660e80bf82352c5db70044254f189f0032d30f8ca9d91d95c670a2bdb160e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c751f98-9133-4738-9052-8dba6020cda6/volumes/kubernetes.io~projected/kube-api-access-skgq6 DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/034cd7c334403750b4defa906b0e09b93223e3e3d5b1a17ec4691b6a2cce323c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02/userdata/shm DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c7c8096-81d8-4c14-b102-08569d25da31/volumes/kubernetes.io~projected/kube-api-access-2pc8z DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34b30d9b6d3b64aa18e694b99251502e5d7fffc7510f82a6415edd2a779ceccd/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8cea9a4e-e914-40b6-9237-345be4aff28c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f23f959-82b3-48cc-8f17-917d8c8c6f6b/volumes/kubernetes.io~projected/kube-api-access-wxc57 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4/userdata/shm DeviceMajor:0 DeviceMinor:54 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4853821b0334d14c66f20cc1ff71c57dbd939417156d20a6f09c401610e96076/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/719b7f16-3b6a-402e-8301-7b498a9ad131/volumes/kubernetes.io~projected/kube-api-access-bknk7 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dee0dda1-07c9-43fc-b7d6-f09b502d0149/volumes/kubernetes.io~projected/kube-api-access-nvv9r DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fee2a93c-fe10-47e5-840c-8d8be03b2715/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0554ee01-1433-4bd7-9a51-d74c757af46a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/915733798ecb455c1a10ae60de736827ff18469f26da3a6cbe871bc903d873b1/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59030007-53c1-42c9-baec-a4d2ec90f3dd/volumes/kubernetes.io~projected/kube-api-access-757tm DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8053559-57b0-468c-a369-893764b0ed4d/volumes/kubernetes.io~projected/kube-api-access-khg96 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8514d431fc512977b1a1605881be8be2bb1b917cfe73bf7e8e3aca8209503a60/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa6cc5e19d492cf2cb49b2106f61c27e5ff4c0d9dc01b6639dc114076e24f040/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7e7e895-264d-4057-a3cc-602abc783f04/volumes/kubernetes.io~projected/kube-api-access-qd54w DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fb0b76f8-eb8d-45ee-afec-b393223b0504/volumes/kubernetes.io~projected/kube-api-access-wzhdx DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fee2a93c-fe10-47e5-840c-8d8be03b2715/volumes/kubernetes.io~projected/kube-api-access-zqrdr DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f7cd28-a6f8-43af-b1dc-f85855564643/volumes/kubernetes.io~projected/kube-api-access-sxhlm DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba969bba97dc07b3af04fa25d4678cbd2f552fc0afa21e7dcebfa4e970a4468c/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/353ac1b3869e6ffb9011f953aaf415aadf741369d91fdd9b0176bece61c6c29a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/979f00a02414c60112e49ee9012cba264a0369fa8653a7e48489ef1c0200110c/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/236577a8-0720-452b-83fa-986cb9aeba14/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/571d12f96a06f1644c4d7dc77e1863e14c5570d657c52f9b2e41f8bad45d3aed/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1499c4e086bed3b3eef48481625653b766960869ddbb32025d24cb8b8d8c45d/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes/kubernetes.io~projected/kube-api-access-ntcw7 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f05d824e-3be1-46f6-8df0-2ade9675e534/volumes/kubernetes.io~projected/kube-api-access-c4sk5 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55d3094e-4769-4135-9878-d0e72089389e/volumes/kubernetes.io~projected/kube-api-access-h4h8n DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:034cd7c33440375 MacAddress:96:7e:af:0b:65:f1 Speed:10000 Mtu:8901} {Name:34b30d9b6d3b64a MacAddress:be:2c:ba:9b:d8:16 Speed:10000 Mtu:8901} {Name:4a2660e80bf8235 MacAddress:82:f3:b4:da:3a:c5 Speed:10000 Mtu:8901} {Name:4b047649a79d2cb MacAddress:6e:21:8f:42:f3:52 Speed:10000 Mtu:8901} {Name:571d12f96a06f16 MacAddress:f2:d2:2a:c6:ae:be Speed:10000 Mtu:8901} {Name:7cfad452ea40a72 MacAddress:1e:4e:87:7b:3e:e8 Speed:10000 Mtu:8901} {Name:8514d431fc51297 MacAddress:02:4e:6b:fd:08:ac Speed:10000 Mtu:8901} {Name:9290f0b0c5766eb MacAddress:ae:26:8d:df:b9:2b Speed:10000 Mtu:8901} {Name:979f00a02414c60 MacAddress:a6:0e:4b:8b:15:fe Speed:10000 Mtu:8901} {Name:a1499c4e086bed3 MacAddress:9a:f4:c3:c2:c7:9b Speed:10000 Mtu:8901} {Name:a27a74f42953674 MacAddress:4e:e0:58:af:13:c7 Speed:10000 Mtu:8901} {Name:ba969bba97dc07b MacAddress:1e:2c:ce:b7:bb:a6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c8:81:c1:c7 Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:2f:16:e3:72:1a Speed:0 Mtu:8901} {Name:c45f4aaf555deb6 MacAddress:ee:8b:d0:d0:10:29 Speed:10000 Mtu:8901} {Name:d0835394a69db45 MacAddress:72:66:64:88:89:fb Speed:10000 Mtu:8901} {Name:d7883d6e70ed6e5 MacAddress:2a:77:92:ed:0f:48 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c8:81:c1:c7 Speed:0 Mtu:9001} {Name:fa6cc5e19d492cf MacAddress:de:65:37:e5:fa:24 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c2:99:4d:f6:78:99 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:32:1f:cb:fa:86:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 12:31:45.881758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.881535 5074 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 12:31:45.881758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.881581 5074 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 12:31:45.881940 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.881917 5074 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 12:31:45.882088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.881940 5074 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-41.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 12:31:45.882129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882090 5074 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 12:31:45.882129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882099 5074 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 12:31:45.882129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882109 5074 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:45.882129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882128 5074 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 12:31:45.882230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882145 5074 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 12:31:45.882319 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882310 5074 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 12:31:45.882429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882420 5074 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 12:31:45.882463 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882456 5074 kubelet.go:493] "Attempting to sync node with API server" Oct 31 12:31:45.882492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882467 5074 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 12:31:45.882492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882476 5074 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 12:31:45.882492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882486 5074 kubelet.go:405] "Adding apiserver pod source" Oct 31 12:31:45.882582 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.882495 5074 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 12:31:45.883583 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.883565 5074 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:31:45.883583 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.883583 5074 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:45.884082 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.884066 5074 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 12:31:45.886345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886331 5074 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 12:31:45.886345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886345 5074 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 12:31:45.886445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886436 5074 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 12:31:45.886628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886617 5074 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 12:31:45.886676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886670 5074 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 12:31:45.886777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886767 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886781 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886789 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886795 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886800 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886815 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886822 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886828 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 12:31:45.886833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886834 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 12:31:45.887230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886840 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 12:31:45.887230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886855 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 12:31:45.887230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886864 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 12:31:45.887230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886891 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 12:31:45.887230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.886898 5074 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 12:31:45.890300 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.890288 5074 server.go:1263] "Started kubelet" Oct 31 12:31:45.890537 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.890363 5074 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 12:31:45.890537 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.890404 5074 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 12:31:45.890537 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.890492 5074 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 12:31:45.891166 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.891145 5074 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 12:31:45.891205 ip-10-0-135-41 systemd[1]: Started Kubernetes Kubelet. Oct 31 12:31:45.895418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.894720 5074 server.go:317] "Adding debug handlers to kubelet server" Oct 31 12:31:45.898600 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.898582 5074 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 12:31:45.898824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.898804 5074 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 12:31:45.898885 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.898854 5074 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 12:31:45.898885 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.898864 5074 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 12:31:45.898980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.898939 5074 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 12:31:45.899840 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.899564 5074 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:26:04 +0000 UTC" deadline="2028-05-01 23:28:41.806166407 +0000 UTC" Oct 31 12:31:45.899840 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.899591 5074 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13162h56m55.906578348s" Oct 31 12:31:45.899840 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:45.899626 5074 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-41.us-east-2.compute.internal\" not found" Oct 31 12:31:45.900807 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.900783 5074 factory.go:153] Registering CRI-O factory Oct 31 12:31:45.900892 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.900810 5074 factory.go:223] Registration of the crio container factory successfully Oct 31 12:31:45.900892 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.900869 5074 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 12:31:45.900892 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.900878 5074 factory.go:55] Registering systemd factory Oct 31 12:31:45.900892 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.900886 5074 factory.go:223] Registration of the systemd container factory successfully Oct 31 12:31:45.901023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.900931 5074 factory.go:103] Registering Raw factory Oct 31 12:31:45.901023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.900953 5074 manager.go:1199] Started watching for new ooms in manager Oct 31 12:31:45.901472 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.901460 5074 manager.go:319] Starting recovery of all containers Oct 31 12:31:45.903059 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903025 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bcc043c-d8ac-4f2a-ae8d-3e233db2842d" volumeName="kubernetes.io/configmap/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-serviceca" seLinuxMountContext="" Oct 31 12:31:45.903145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903068 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bcf0cc-7eef-4cbe-9d72-140932580c56" volumeName="kubernetes.io/secret/55bcf0cc-7eef-4cbe-9d72-140932580c56-serving-cert" seLinuxMountContext="" Oct 31 12:31:45.903145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903081 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c751f98-9133-4738-9052-8dba6020cda6" volumeName="kubernetes.io/empty-dir/8c751f98-9133-4738-9052-8dba6020cda6-tmp-dir" seLinuxMountContext="" Oct 31 12:31:45.903145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903092 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a069e898-8264-4573-b885-2528dadb0305" volumeName="kubernetes.io/empty-dir/a069e898-8264-4573-b885-2528dadb0305-tmp-dir" seLinuxMountContext="" Oct 31 12:31:45.903145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903113 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4942177-472e-47cf-934b-92b58447dfae" volumeName="kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token" seLinuxMountContext="" Oct 31 12:31:45.903145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903125 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4942177-472e-47cf-934b-92b58447dfae" volumeName="kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets" seLinuxMountContext="" Oct 31 12:31:45.903145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903138 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bcf0cc-7eef-4cbe-9d72-140932580c56" volumeName="kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903166 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bcf0cc-7eef-4cbe-9d72-140932580c56" volumeName="kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-service-ca-bundle" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903197 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55d3094e-4769-4135-9878-d0e72089389e" volumeName="kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903208 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e7e895-264d-4057-a3cc-602abc783f04" volumeName="kubernetes.io/configmap/d7e7e895-264d-4057-a3cc-602abc783f04-signing-cabundle" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903233 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300d5937-06ea-46c4-894e-17ab3aea7815" volumeName="kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903245 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c751f98-9133-4738-9052-8dba6020cda6" volumeName="kubernetes.io/configmap/8c751f98-9133-4738-9052-8dba6020cda6-config-volume" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903265 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c751f98-9133-4738-9052-8dba6020cda6" volumeName="kubernetes.io/projected/8c751f98-9133-4738-9052-8dba6020cda6-kube-api-access-skgq6" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903290 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="418e7a93-3e5c-41d8-a0be-bf74b3919943" volumeName="kubernetes.io/projected/418e7a93-3e5c-41d8-a0be-bf74b3919943-kube-api-access-9l8rb" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903304 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c7c8096-81d8-4c14-b102-08569d25da31" volumeName="kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-cni-binary-copy" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903344 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f7cd28-a6f8-43af-b1dc-f85855564643" volumeName="kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-tmp" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903381 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fee2a93c-fe10-47e5-840c-8d8be03b2715" volumeName="kubernetes.io/projected/fee2a93c-fe10-47e5-840c-8d8be03b2715-kube-api-access-zqrdr" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903397 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300d5937-06ea-46c4-894e-17ab3aea7815" volumeName="kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903409 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bcf0cc-7eef-4cbe-9d72-140932580c56" volumeName="kubernetes.io/projected/55bcf0cc-7eef-4cbe-9d72-140932580c56-kube-api-access-p9r6x" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903431 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746d043e-1f60-464f-ba50-6a437eb9089f" volumeName="kubernetes.io/configmap/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-config" seLinuxMountContext="" Oct 31 12:31:45.903450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903452 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4942177-472e-47cf-934b-92b58447dfae" volumeName="kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903464 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c765c0f2-1bc6-4eee-9fde-317749984f5c" volumeName="kubernetes.io/projected/c765c0f2-1bc6-4eee-9fde-317749984f5c-kube-api-access-w2ppx" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903505 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e7e895-264d-4057-a3cc-602abc783f04" volumeName="kubernetes.io/secret/d7e7e895-264d-4057-a3cc-602abc783f04-signing-key" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903521 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb0b76f8-eb8d-45ee-afec-b393223b0504" volumeName="kubernetes.io/projected/fb0b76f8-eb8d-45ee-afec-b393223b0504-kube-api-access-wzhdx" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903553 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fee2a93c-fe10-47e5-840c-8d8be03b2715" volumeName="kubernetes.io/secret/fee2a93c-fe10-47e5-840c-8d8be03b2715-serving-cert" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903580 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236577a8-0720-452b-83fa-986cb9aeba14" volumeName="kubernetes.io/projected/236577a8-0720-452b-83fa-986cb9aeba14-kube-api-access-mq2fm" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903593 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236577a8-0720-452b-83fa-986cb9aeba14" volumeName="kubernetes.io/secret/236577a8-0720-452b-83fa-986cb9aeba14-serving-cert" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903608 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bcf0cc-7eef-4cbe-9d72-140932580c56" volumeName="kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-snapshots" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903706 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="698d7a34-d950-442b-b12e-b0954c8fe873" volumeName="kubernetes.io/configmap/698d7a34-d950-442b-b12e-b0954c8fe873-nginx-conf" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903720 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746d043e-1f60-464f-ba50-6a437eb9089f" volumeName="kubernetes.io/projected/746d043e-1f60-464f-ba50-6a437eb9089f-kube-api-access-6dxwp" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903732 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf409f2b-9f95-4f84-bbd6-d9cc841fc074" volumeName="kubernetes.io/projected/bf409f2b-9f95-4f84-bbd6-d9cc841fc074-kube-api-access-j5nm4" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903756 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d409cf23-c109-4dfb-a569-5041531269dc" volumeName="kubernetes.io/projected/d409cf23-c109-4dfb-a569-5041531269dc-kube-api-access-99f5v" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903778 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236577a8-0720-452b-83fa-986cb9aeba14" volumeName="kubernetes.io/configmap/236577a8-0720-452b-83fa-986cb9aeba14-config" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903791 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f23f959-82b3-48cc-8f17-917d8c8c6f6b" volumeName="kubernetes.io/projected/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kube-api-access-wxc57" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903813 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="579b4046-76c9-4d7e-b77a-9888510a22a8" volumeName="kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-stats-auth" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903826 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59030007-53c1-42c9-baec-a4d2ec90f3dd" volumeName="kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903840 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e7e895-264d-4057-a3cc-602abc783f04" volumeName="kubernetes.io/projected/d7e7e895-264d-4057-a3cc-602abc783f04-kube-api-access-qd54w" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903867 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e60a880f-7773-41dd-9995-3b1719fe8878" volumeName="kubernetes.io/empty-dir/e60a880f-7773-41dd-9995-3b1719fe8878-data-volume" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903880 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59030007-53c1-42c9-baec-a4d2ec90f3dd" volumeName="kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-binary-copy" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903890 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c7c8096-81d8-4c14-b102-08569d25da31" volumeName="kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-multus-daemon-config" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903905 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e040a356-8196-40c6-8f2c-a94bb4c6bfe4" volumeName="kubernetes.io/projected/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-kube-api-access-zxjsf" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903918 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300d5937-06ea-46c4-894e-17ab3aea7815" volumeName="kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903930 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300d5937-06ea-46c4-894e-17ab3aea7815" volumeName="kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903942 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59030007-53c1-42c9-baec-a4d2ec90f3dd" volumeName="kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903964 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cea9a4e-e914-40b6-9237-345be4aff28c" volumeName="kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903978 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fee2a93c-fe10-47e5-840c-8d8be03b2715" volumeName="kubernetes.io/configmap/fee2a93c-fe10-47e5-840c-8d8be03b2715-config" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.903993 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0554ee01-1433-4bd7-9a51-d74c757af46a" volumeName="kubernetes.io/configmap/0554ee01-1433-4bd7-9a51-d74c757af46a-konnectivity-ca" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904005 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300d5937-06ea-46c4-894e-17ab3aea7815" volumeName="kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904018 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8a0aad-3971-455a-8fda-f40c28d8bc13" volumeName="kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-script-lib" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904031 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="579b4046-76c9-4d7e-b77a-9888510a22a8" volumeName="kubernetes.io/projected/579b4046-76c9-4d7e-b77a-9888510a22a8-kube-api-access-8pmcf" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904044 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59030007-53c1-42c9-baec-a4d2ec90f3dd" volumeName="kubernetes.io/projected/59030007-53c1-42c9-baec-a4d2ec90f3dd-kube-api-access-757tm" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904058 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a069e898-8264-4573-b885-2528dadb0305" volumeName="kubernetes.io/projected/a069e898-8264-4573-b885-2528dadb0305-kube-api-access-6pqtb" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904070 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4942177-472e-47cf-934b-92b58447dfae" volumeName="kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904085 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4942177-472e-47cf-934b-92b58447dfae" volumeName="kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904099 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300d5937-06ea-46c4-894e-17ab3aea7815" volumeName="kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted" seLinuxMountContext="" Oct 31 12:31:45.904150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904111 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8a0aad-3971-455a-8fda-f40c28d8bc13" volumeName="kubernetes.io/projected/4a8a0aad-3971-455a-8fda-f40c28d8bc13-kube-api-access-fv66w" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904181 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8053559-57b0-468c-a369-893764b0ed4d" volumeName="kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-config" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904199 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8053559-57b0-468c-a369-893764b0ed4d" volumeName="kubernetes.io/projected/a8053559-57b0-468c-a369-893764b0ed4d-kube-api-access-khg96" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904213 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e60a880f-7773-41dd-9995-3b1719fe8878" volumeName="kubernetes.io/projected/e60a880f-7773-41dd-9995-3b1719fe8878-kube-api-access-fzbjm" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904226 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300d5937-06ea-46c4-894e-17ab3aea7815" volumeName="kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904239 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8a0aad-3971-455a-8fda-f40c28d8bc13" volumeName="kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-env-overrides" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904252 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="579b4046-76c9-4d7e-b77a-9888510a22a8" volumeName="kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-default-certificate" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904265 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b057b6d-b9d1-4642-95f3-41ba5498c24d" volumeName="kubernetes.io/projected/5b057b6d-b9d1-4642-95f3-41ba5498c24d-kube-api-access-fgbcf" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904278 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="719b7f16-3b6a-402e-8301-7b498a9ad131" volumeName="kubernetes.io/projected/719b7f16-3b6a-402e-8301-7b498a9ad131-kube-api-access-bknk7" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904291 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8053559-57b0-468c-a369-893764b0ed4d" volumeName="kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-trusted-ca" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904304 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8d6247b-578f-4ee3-8fb4-01bb7da39a7d" volumeName="kubernetes.io/projected/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d-kube-api-access-s7tr4" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904317 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bcc043c-d8ac-4f2a-ae8d-3e233db2842d" volumeName="kubernetes.io/projected/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-kube-api-access-n7tbp" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904331 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55d3094e-4769-4135-9878-d0e72089389e" volumeName="kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904343 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c7c8096-81d8-4c14-b102-08569d25da31" volumeName="kubernetes.io/projected/9c7c8096-81d8-4c14-b102-08569d25da31-kube-api-access-2pc8z" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904355 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4942177-472e-47cf-934b-92b58447dfae" volumeName="kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904368 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8053559-57b0-468c-a369-893764b0ed4d" volumeName="kubernetes.io/secret/a8053559-57b0-468c-a369-893764b0ed4d-serving-cert" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904382 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d409cf23-c109-4dfb-a569-5041531269dc" volumeName="kubernetes.io/configmap/d409cf23-c109-4dfb-a569-5041531269dc-iptables-alerter-script" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904395 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e040a356-8196-40c6-8f2c-a94bb4c6bfe4" volumeName="kubernetes.io/secret/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-cert" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904409 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f05d824e-3be1-46f6-8df0-2ade9675e534" volumeName="kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904421 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4942177-472e-47cf-934b-92b58447dfae" volumeName="kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904435 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bcf0cc-7eef-4cbe-9d72-140932580c56" volumeName="kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-tmp" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904449 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8a0aad-3971-455a-8fda-f40c28d8bc13" volumeName="kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-config" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904462 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55d3094e-4769-4135-9878-d0e72089389e" volumeName="kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904477 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee0dda1-07c9-43fc-b7d6-f09b502d0149" volumeName="kubernetes.io/configmap/dee0dda1-07c9-43fc-b7d6-f09b502d0149-telemetry-config" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904489 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee0dda1-07c9-43fc-b7d6-f09b502d0149" volumeName="kubernetes.io/projected/dee0dda1-07c9-43fc-b7d6-f09b502d0149-kube-api-access-nvv9r" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904502 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f7cd28-a6f8-43af-b1dc-f85855564643" volumeName="kubernetes.io/projected/f0f7cd28-a6f8-43af-b1dc-f85855564643-kube-api-access-sxhlm" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904516 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0554ee01-1433-4bd7-9a51-d74c757af46a" volumeName="kubernetes.io/secret/0554ee01-1433-4bd7-9a51-d74c757af46a-agent-certs" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904528 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8a0aad-3971-455a-8fda-f40c28d8bc13" volumeName="kubernetes.io/secret/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904541 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55d3094e-4769-4135-9878-d0e72089389e" volumeName="kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904553 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9" volumeName="kubernetes.io/projected/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9-kube-api-access-fdgb7" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904565 5074 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f7cd28-a6f8-43af-b1dc-f85855564643" volumeName="kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-tuned" seLinuxMountContext="" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904577 5074 reconstruct.go:97] "Volume reconstruction finished" Oct 31 12:31:45.905142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.904585 5074 reconciler.go:29] "Reconciler: start to sync state" Oct 31 12:31:45.909619 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.909471 5074 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:45.912129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.912104 5074 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:45.913517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.913495 5074 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 12:31:45.915532 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.915006 5074 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:45.925526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.925304 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/1.log" Oct 31 12:31:45.925526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.925436 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/1.log" Oct 31 12:31:45.928727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.928680 5074 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 12:31:45.930172 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.930151 5074 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 12:31:45.930172 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.930173 5074 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 12:31:45.930327 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.930194 5074 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 12:31:45.930327 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:45.930229 5074 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 12:31:45.934459 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.934441 5074 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 12:31:45.946015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.945986 5074 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="6230b924d6c7d1583743bbb3775310712d7b2e70b911fdc157a03b6c3579ce51" exitCode=0 Oct 31 12:31:45.946015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.946014 5074 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="c81841a1450b9168278a6504581c2bd88ad7ac6e890a53cfe8c07f0d3631ede3" exitCode=0 Oct 31 12:31:45.946146 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.946025 5074 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="f1c581db68b380f5e00106fa2ddeaf81dc7d6691e3dca743a85b274db64a795b" exitCode=0 Oct 31 12:31:45.946146 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.946031 5074 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="fed21df9ed8b0a30361494cfc19c5e279ff1b6f55e24e81ac5e00b045a1f499f" exitCode=0 Oct 31 12:31:45.946146 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.946039 5074 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="0831b148fcfdf420ad52902629556a0e2b459bfd1f63e19380d789db03b08601" exitCode=0 Oct 31 12:31:45.946146 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.946046 5074 generic.go:356] "Generic (PLEG): container finished" podID="59030007-53c1-42c9-baec-a4d2ec90f3dd" containerID="60eae7931cf1ad525d2bfb83395db0bbc8f706d1675774329f2a3e159cb41d60" exitCode=0 Oct 31 12:31:45.953096 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.953075 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/1.log" Oct 31 12:31:45.953205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.953103 5074 generic.go:356] "Generic (PLEG): container finished" podID="a8053559-57b0-468c-a369-893764b0ed4d" containerID="67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692" exitCode=255 Oct 31 12:31:45.961475 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:45.961447 5074 generic.go:356] "Generic (PLEG): container finished" podID="7397e52f9425481ec78d95c85d33a100" containerID="569d1999e691fe21bf82eaddbfdd3963d6ecb02badf5a7ef84f352ab5f2205f0" exitCode=0 Oct 31 12:31:46.000471 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.000447 5074 manager.go:324] Recovery completed Oct 31 12:31:46.022965 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.022896 5074 cpu_manager.go:226] "Starting" policy="none" Oct 31 12:31:46.022965 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.022920 5074 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 12:31:46.022965 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.022942 5074 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 12:31:46.023154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023092 5074 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 12:31:46.023154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023109 5074 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 12:31:46.023154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023131 5074 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 12:31:46.023154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023139 5074 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 12:31:46.023154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023146 5074 policy_none.go:50] "Start" Oct 31 12:31:46.023285 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023157 5074 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 12:31:46.023285 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023170 5074 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 12:31:46.023336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023307 5074 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 12:31:46.023336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023315 5074 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 12:31:46.023336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.023321 5074 policy_none.go:44] "Start" Oct 31 12:31:46.027745 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.027715 5074 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 12:31:46.030199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.030187 5074 manager.go:341] "Starting Device Plugin manager" Oct 31 12:31:46.030318 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:46.030291 5074 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 12:31:46.030737 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:46.030212 5074 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 12:31:46.030805 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.030740 5074 server.go:87] "Starting device plugin registration server" Oct 31 12:31:46.030990 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.030979 5074 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 12:31:46.031022 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.030991 5074 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 12:31:46.031106 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.031086 5074 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 12:31:46.031237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.031195 5074 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 12:31:46.031300 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.031251 5074 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 12:31:46.031300 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.031259 5074 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 12:31:46.031389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.031309 5074 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:31:46.031234376Z","UUID":"6c70f606-9b23-4c0d-97e5-ecb46f590e3b","Handler":null,"Name":"","Endpoint":""} Oct 31 12:31:46.033074 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.033052 5074 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 12:31:46.033173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.033085 5074 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 12:31:46.033173 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:46.033106 5074 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 12:31:46.131508 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.131473 5074 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 12:31:46.132980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.132955 5074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 12:31:46.133089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.132989 5074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 12:31:46.133089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.133003 5074 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 12:31:46.133089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.133045 5074 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.149485 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.149459 5074 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.149580 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.149526 5074 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.230619 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.230569 5074 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal"] Oct 31 12:31:46.231553 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.231514 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"7397e52f9425481ec78d95c85d33a100","Type":"ContainerStarted","Data":"763139fd980070db425f6480f037647a6d030d2679ef172db7d0761d2fc361b8"} Oct 31 12:31:46.231666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.231557 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"7397e52f9425481ec78d95c85d33a100","Type":"ContainerDied","Data":"569d1999e691fe21bf82eaddbfdd3963d6ecb02badf5a7ef84f352ab5f2205f0"} Oct 31 12:31:46.231666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.231570 5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.231666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.231572 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"7397e52f9425481ec78d95c85d33a100","Type":"ContainerStarted","Data":"b39b8ad3ca6734c068e68db652947026b9b950ef46a5ca2d44934f669d579af4"} Oct 31 12:31:46.231788 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.231758 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"f84469bc02ae1e7d219895e3165c40bd","Type":"ContainerStarted","Data":"c6e42bbcea2a8e54271a477416a7b0b8a29dd7f9f2057cda4a8fe3a5edbcebe3"} Oct 31 12:31:46.231872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.231790 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" event={"ID":"f84469bc02ae1e7d219895e3165c40bd","Type":"ContainerStarted","Data":"842186a8dfbb58d0613b109696d781641f307180710eed935425c316a420ab02"} Oct 31 12:31:46.231872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.231827 5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.246115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.246096 5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:46.246171 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:46.246134 5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.246977 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.246965 5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:46.247013 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:46.246998 5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.315404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.315285 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.315404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.315393 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.315545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.315416 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84469bc02ae1e7d219895e3165c40bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"f84469bc02ae1e7d219895e3165c40bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.415887 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.415856 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.415887 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.415888 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.416053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.415910 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84469bc02ae1e7d219895e3165c40bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"f84469bc02ae1e7d219895e3165c40bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.416053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.415951 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84469bc02ae1e7d219895e3165c40bd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"f84469bc02ae1e7d219895e3165c40bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.416053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.415970 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.416053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.415972 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7397e52f9425481ec78d95c85d33a100-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" (UID: \"7397e52f9425481ec78d95c85d33a100\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:46.882922 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.882873 5074 apiserver.go:52] "Watching apiserver" Oct 31 12:31:46.896387 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.896357 5074 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 12:31:46.897089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.897061 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4","openshift-dns/node-resolver-9v4bh","openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf","openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5","openshift-console-operator/console-operator-7cd56b8dbc-7vnxg","openshift-image-registry/image-registry-85dd98c8f4-j6vtq","openshift-multus/network-metrics-daemon-5kl4n","openshift-image-registry/image-registry-6f8479bcff-g6cvb","kube-system/kas-connection-checker-685895c7c9-6ps96","kube-system/kas-connection-checker-685895c7c9-tn6pm","openshift-insights/insights-runtime-extractor-jc7gp","openshift-image-registry/node-ca-qkx64","openshift-ingress/router-default-865bf4d8bc-d5gwq","openshift-network-diagnostics/network-check-target-gjvxs","openshift-network-console/networking-console-plugin-86c478bb88-pnzjp","openshift-network-operator/iptables-alerter-6hdft","openshift-cluster-node-tuning-operator/tuned-7z7sx","kube-system/konnectivity-agent-b4cc4","openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9","openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc","openshift-multus/multus-additional-cni-plugins-9tkzk","kube-system/kas-connection-checker-685895c7c9-hxg8d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6","openshift-ingress-canary/ingress-canary-bqcvc","openshift-insights/periodic-gathering-7vhj5-f2fsm","openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2","kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal","openshift-insights/insights-operator-7c4986d449-tc2tp","openshift-multus/multus-db7cd","kube-system/global-pull-secret-syncer-sk7rm","openshift-dns/dns-default-p4psl","openshift-service-ca/service-ca-5fffc97bdf-sjz88","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8","openshift-ovn-kubernetes/ovnkube-node-r8m9k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal"] Oct 31 12:31:46.898126 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.898105 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:46.899071 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.899051 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:46.900202 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.900185 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:46.901369 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.901348 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:46.901812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.901790 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:46.904700 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.904678 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.904972 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.904952 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.905079 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.905042 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.905499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.905479 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 12:31:46.905684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.905662 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.905766 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.905687 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 12:31:46.905766 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.905696 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.905766 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.905756 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nwqs2\"" Oct 31 12:31:46.906029 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906011 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.906115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906063 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 12:31:46.906115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906100 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.906115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906111 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906115 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906156 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906171 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906181 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906172 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906212 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906234 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 12:31:46.906259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906237 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-29qxx\"" Oct 31 12:31:46.906495 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.906310 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 12:31:46.907543 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.907525 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.907661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.907600 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:46.908354 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.908335 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-r82tg\"" Oct 31 12:31:46.908708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.908448 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5tdjb\"" Oct 31 12:31:46.909061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.908503 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mdmxf\"" Oct 31 12:31:46.909167 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.908779 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 12:31:46.909227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.908861 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pnqgm\"" Oct 31 12:31:46.909280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.908976 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 12:31:46.909326 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.909073 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vbv96\"" Oct 31 12:31:46.911138 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.911116 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 12:31:46.911835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.911813 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 12:31:46.911978 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.911921 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 12:31:46.912080 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.912060 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 12:31:46.912983 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.912950 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 12:31:46.912983 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.912964 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 12:31:46.913171 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.913036 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xwjnx\"" Oct 31 12:31:46.913171 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.913107 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:46.913391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.913352 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 12:31:46.913391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.913380 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nc6nm\"" Oct 31 12:31:46.913507 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.913388 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cclp9\"" Oct 31 12:31:46.914192 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.914170 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 12:31:46.914192 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.914180 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 12:31:46.914322 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.914200 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 12:31:46.914322 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.914245 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 12:31:46.915058 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915042 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:46.915233 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915216 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.915352 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915337 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.915419 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915348 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.915472 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915343 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 12:31:46.915530 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915476 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.915530 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915353 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.915618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915578 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 12:31:46.915618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915587 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.915618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915616 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915590 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915647 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915681 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915650 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w8vnc\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915681 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9dfhb\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915683 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915715 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915719 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915747 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.915782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915780 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7phhl\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915792 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bkb2g\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915796 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915825 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915846 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915792 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915864 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915856 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915873 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915893 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915897 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8c8p7\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915906 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915940 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915947 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915955 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915802 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.915962 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.916132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916082 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:46.916902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916880 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 12:31:46.916902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916898 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 12:31:46.917011 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916912 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvpb\"" Oct 31 12:31:46.917011 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916934 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 12:31:46.917011 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916934 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 12:31:46.917011 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916935 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l5d6v\"" Oct 31 12:31:46.917011 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.916955 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 12:31:46.917151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917017 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hk7cx\"" Oct 31 12:31:46.917151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917060 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.917341 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917321 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmbl2\"" Oct 31 12:31:46.917399 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917378 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zfd9q\"" Oct 31 12:31:46.917834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917815 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ppx\" (UniqueName: \"kubernetes.io/projected/c765c0f2-1bc6-4eee-9fde-317749984f5c-kube-api-access-w2ppx\") pod \"kas-connection-checker-685895c7c9-hxg8d\" (UID: \"c765c0f2-1bc6-4eee-9fde-317749984f5c\") " pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" Oct 31 12:31:46.917934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917850 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.917934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917877 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khg96\" (UniqueName: \"kubernetes.io/projected/a8053559-57b0-468c-a369-893764b0ed4d-kube-api-access-khg96\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.917934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917902 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fm\" (UniqueName: \"kubernetes.io/projected/236577a8-0720-452b-83fa-986cb9aeba14-kube-api-access-mq2fm\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:46.918092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.917954 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgbcf\" (UniqueName: \"kubernetes.io/projected/5b057b6d-b9d1-4642-95f3-41ba5498c24d-kube-api-access-fgbcf\") pod \"kas-connection-checker-685895c7c9-6ps96\" (UID: \"5b057b6d-b9d1-4642-95f3-41ba5498c24d\") " pod="kube-system/kas-connection-checker-685895c7c9-6ps96" Oct 31 12:31:46.918092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918009 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:46.918092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918040 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-config\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.918092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918066 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8053559-57b0-468c-a369-893764b0ed4d-serving-cert\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918094 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bknk7\" (UniqueName: \"kubernetes.io/projected/719b7f16-3b6a-402e-8301-7b498a9ad131-kube-api-access-bknk7\") pod \"volume-data-source-validator-58c9f5f46d-sxrj4\" (UID: \"719b7f16-3b6a-402e-8301-7b498a9ad131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918143 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dee0dda1-07c9-43fc-b7d6-f09b502d0149-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918169 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvv9r\" (UniqueName: \"kubernetes.io/projected/dee0dda1-07c9-43fc-b7d6-f09b502d0149-kube-api-access-nvv9r\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918197 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tr4\" (UniqueName: \"kubernetes.io/projected/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d-kube-api-access-s7tr4\") pod \"kas-connection-checker-685895c7c9-tn6pm\" (UID: \"e8d6247b-578f-4ee3-8fb4-01bb7da39a7d\") " pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918225 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918238 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918251 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:46.918286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918284 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxwp\" (UniqueName: \"kubernetes.io/projected/746d043e-1f60-464f-ba50-6a437eb9089f-kube-api-access-6dxwp\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:46.918653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918308 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236577a8-0720-452b-83fa-986cb9aeba14-serving-cert\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:46.918653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918333 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236577a8-0720-452b-83fa-986cb9aeba14-config\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:46.918653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918338 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8053559-57b0-468c-a369-893764b0ed4d-config\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.918653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918420 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dee0dda1-07c9-43fc-b7d6-f09b502d0149-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:46.918653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918523 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8053559-57b0-468c-a369-893764b0ed4d-serving-cert\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.918653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918534 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/236577a8-0720-452b-83fa-986cb9aeba14-config\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:46.918653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918606 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:46.918980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918713 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/236577a8-0720-452b-83fa-986cb9aeba14-serving-cert\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:46.918980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.918882 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 12:31:46.919237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919193 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k457\"" Oct 31 12:31:46.919237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919230 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 12:31:46.919352 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919319 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l77g8\"" Oct 31 12:31:46.919385 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919365 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zbx6v\"" Oct 31 12:31:46.919519 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919502 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.919607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919591 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 12:31:46.919688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919541 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.919688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919549 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 12:31:46.919688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919627 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nthx6\"" Oct 31 12:31:46.919688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919576 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 12:31:46.919688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919673 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 12:31:46.919688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919508 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 12:31:46.919871 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.919581 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrn76\"" Oct 31 12:31:46.920299 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.920282 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 12:31:46.920299 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.920287 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 12:31:46.920299 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.920300 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52dg7\"" Oct 31 12:31:46.922503 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.922472 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.922503 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.922487 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.922627 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.922534 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ttjjk\"" Oct 31 12:31:46.923664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.923645 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 12:31:46.923737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.923713 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9rdmw\"" Oct 31 12:31:46.924541 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.924527 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 12:31:46.924541 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.924534 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 12:31:46.924811 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.924798 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 12:31:46.928864 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.928845 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ppx\" (UniqueName: \"kubernetes.io/projected/c765c0f2-1bc6-4eee-9fde-317749984f5c-kube-api-access-w2ppx\") pod \"kas-connection-checker-685895c7c9-hxg8d\" (UID: \"c765c0f2-1bc6-4eee-9fde-317749984f5c\") " pod="kube-system/kas-connection-checker-685895c7c9-hxg8d" Oct 31 12:31:46.929189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.929169 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2fm\" (UniqueName: \"kubernetes.io/projected/236577a8-0720-452b-83fa-986cb9aeba14-kube-api-access-mq2fm\") pod \"service-ca-operator-544d749c8c-nggm9\" (UID: \"236577a8-0720-452b-83fa-986cb9aeba14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" Oct 31 12:31:46.930031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.930016 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvv9r\" (UniqueName: \"kubernetes.io/projected/dee0dda1-07c9-43fc-b7d6-f09b502d0149-kube-api-access-nvv9r\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:46.930861 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.930839 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khg96\" (UniqueName: \"kubernetes.io/projected/a8053559-57b0-468c-a369-893764b0ed4d-kube-api-access-khg96\") pod \"console-operator-7cd56b8dbc-7vnxg\" (UID: \"a8053559-57b0-468c-a369-893764b0ed4d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:46.931668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.931650 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgbcf\" (UniqueName: \"kubernetes.io/projected/5b057b6d-b9d1-4642-95f3-41ba5498c24d-kube-api-access-fgbcf\") pod \"kas-connection-checker-685895c7c9-6ps96\" (UID: \"5b057b6d-b9d1-4642-95f3-41ba5498c24d\") " pod="kube-system/kas-connection-checker-685895c7c9-6ps96" Oct 31 12:31:46.935933 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.935914 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tr4\" (UniqueName: \"kubernetes.io/projected/e8d6247b-578f-4ee3-8fb4-01bb7da39a7d-kube-api-access-s7tr4\") pod \"kas-connection-checker-685895c7c9-tn6pm\" (UID: \"e8d6247b-578f-4ee3-8fb4-01bb7da39a7d\") " pod="kube-system/kas-connection-checker-685895c7c9-tn6pm" Oct 31 12:31:46.936000 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.935938 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bknk7\" (UniqueName: \"kubernetes.io/projected/719b7f16-3b6a-402e-8301-7b498a9ad131-kube-api-access-bknk7\") pod \"volume-data-source-validator-58c9f5f46d-sxrj4\" (UID: \"719b7f16-3b6a-402e-8301-7b498a9ad131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sxrj4" Oct 31 12:31:46.939383 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:46.939366 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxwp\" (UniqueName: \"kubernetes.io/projected/746d043e-1f60-464f-ba50-6a437eb9089f-kube-api-access-6dxwp\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:47.000783 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.000760 5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:47.000953 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.000937 5074 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:47.001733 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.001718 5074 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 12:31:47.012872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.012857 5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:47.012932 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.012893 5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:47.018831 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.018812 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmcf\" (UniqueName: \"kubernetes.io/projected/579b4046-76c9-4d7e-b77a-9888510a22a8-kube-api-access-8pmcf\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.018872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.018851 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.018910 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.018877 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-system-cni-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.018941 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.018905 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhdx\" (UniqueName: \"kubernetes.io/projected/fb0b76f8-eb8d-45ee-afec-b393223b0504-kube-api-access-wzhdx\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:47.019057 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019033 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.019119 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019078 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409cf23-c109-4dfb-a569-5041531269dc-host-slash\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:47.019119 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019105 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-systemd\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.019184 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019170 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-netd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.019227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019212 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-kubernetes\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.019347 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019332 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-k8s-cni-cncf-io\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.019399 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019362 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-script-lib\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.019399 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019390 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r6x\" (UniqueName: \"kubernetes.io/projected/55bcf0cc-7eef-4cbe-9d72-140932580c56-kube-api-access-p9r6x\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.019490 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019417 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:47.019490 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019446 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.019576 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019503 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5nm4\" (UniqueName: \"kubernetes.io/projected/bf409f2b-9f95-4f84-bbd6-d9cc841fc074-kube-api-access-j5nm4\") pod \"network-check-source-6f78d59f99-62jh5\" (UID: \"bf409f2b-9f95-4f84-bbd6-d9cc841fc074\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" Oct 31 12:31:47.019576 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019535 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-systemd-units\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.019576 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019559 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019576 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqtb\" (UniqueName: \"kubernetes.io/projected/a069e898-8264-4573-b885-2528dadb0305-kube-api-access-6pqtb\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019588 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-script-lib\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019601 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-kubelet-config\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019658 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysconfig\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019694 5074 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019697 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgq6\" (UniqueName: \"kubernetes.io/projected/8c751f98-9133-4738-9052-8dba6020cda6-kube-api-access-skgq6\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019723 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.019748 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.019733 5074 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-41.us-east-2.compute.internal" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019755 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-tmp\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019805 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd54w\" (UniqueName: \"kubernetes.io/projected/d7e7e895-264d-4057-a3cc-602abc783f04-kube-api-access-qd54w\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019838 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0554ee01-1433-4bd7-9a51-d74c757af46a-konnectivity-ca\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019867 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019881 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-tmp\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.019896 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020025 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-kubelet\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.020065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020052 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-var-lib-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020082 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020129 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-sys\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020163 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fee2a93c-fe10-47e5-840c-8d8be03b2715-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.020168 5074 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020194 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.020259 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.520240169 +0000 UTC m=+1.697958172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020270 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0554ee01-1433-4bd7-9a51-d74c757af46a-konnectivity-ca\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020301 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020323 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-cnibin\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.020404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020357 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020417 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-multus-daemon-config\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020430 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fee2a93c-fe10-47e5-840c-8d8be03b2715-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020443 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-netns\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020570 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020573 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020618 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020665 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-var-lib-kubelet\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020699 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrdr\" (UniqueName: \"kubernetes.io/projected/fee2a93c-fe10-47e5-840c-8d8be03b2715-kube-api-access-zqrdr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:47.020740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020733 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-socket-dir-parent\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.020976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020744 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-multus-daemon-config\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.020976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020833 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.020976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020876 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-bin\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.020976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020898 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.020976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020930 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.020976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020951 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:47.020976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020968 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-tmp\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.021189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.020983 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.021189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021051 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-tmp\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.021189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021104 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.021189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021133 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-node-log\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.021189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021167 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-kubelet\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.021373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021195 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.021373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021233 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-modprobe-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.021373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021280 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-etc-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.021373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021309 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.021373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021319 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-log-socket\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.021373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021346 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a069e898-8264-4573-b885-2528dadb0305-hosts-file\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:47.021373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021371 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.021557 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021398 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-host\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:47.021557 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021496 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.021557 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021522 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.021557 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021528 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-conf\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.021685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021555 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-run\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.021685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021581 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-cni-binary-copy\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.021685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021669 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.021787 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021711 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-stats-auth\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.021787 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021723 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.021877 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021795 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c7c8096-81d8-4c14-b102-08569d25da31-cni-binary-copy\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.021877 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021821 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.021877 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021848 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-stats-auth\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021850 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021917 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-cnibin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021921 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021941 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-etc-kubernetes\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021972 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-cert\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022015 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.021998 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-netns\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022099 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-serviceca\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022137 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022141 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bcf0cc-7eef-4cbe-9d72-140932580c56-serving-cert\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022198 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-cert\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:47.022256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022244 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022284 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069e898-8264-4573-b885-2528dadb0305-tmp-dir\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022311 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovn-node-metrics-cert\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022337 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7tbp\" (UniqueName: \"kubernetes.io/projected/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-kube-api-access-n7tbp\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022365 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069e898-8264-4573-b885-2528dadb0305-tmp-dir\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022391 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022433 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022517 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovn-node-metrics-cert\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022534 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022563 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-default-certificate\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022614 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fee2a93c-fe10-47e5-840c-8d8be03b2715-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022616 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bcf0cc-7eef-4cbe-9d72-140932580c56-serving-cert\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022674 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-serviceca\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022712 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bcf0cc-7eef-4cbe-9d72-140932580c56-service-ca-bundle\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.022800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022782 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-tuned\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022813 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-default-certificate\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022844 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4h8n\" (UniqueName: \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022886 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-tuned\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022917 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-conf-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022941 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-sys-fs\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022969 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxc57\" (UniqueName: \"kubernetes.io/projected/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kube-api-access-wxc57\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.022985 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fee2a93c-fe10-47e5-840c-8d8be03b2715-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023009 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv66w\" (UniqueName: \"kubernetes.io/projected/4a8a0aad-3971-455a-8fda-f40c28d8bc13-kube-api-access-fv66w\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023177 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023201 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-registration-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023219 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757tm\" (UniqueName: \"kubernetes.io/projected/59030007-53c1-42c9-baec-a4d2ec90f3dd-kube-api-access-757tm\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.023243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023236 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023272 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-host\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023300 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/f0f7cd28-a6f8-43af-b1dc-f85855564643-kube-api-access-sxhlm\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023326 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023422 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023465 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d409cf23-c109-4dfb-a569-5041531269dc-iptables-alerter-script\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023476 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023488 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023505 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-system-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023551 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-socket-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023571 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-device-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023595 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntcw7\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023602 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59030007-53c1-42c9-baec-a4d2ec90f3dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023600 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023650 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c751f98-9133-4738-9052-8dba6020cda6-tmp-dir\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023717 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.023781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023754 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d409cf23-c109-4dfb-a569-5041531269dc-iptables-alerter-script\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023790 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c751f98-9133-4738-9052-8dba6020cda6-tmp-dir\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023821 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023840 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e60a880f-7773-41dd-9995-3b1719fe8878-crio-socket\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023855 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023891 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-lib-modules\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023923 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8rb\" (UniqueName: \"kubernetes.io/projected/418e7a93-3e5c-41d8-a0be-bf74b3919943-kube-api-access-9l8rb\") pod \"migrator-66df5d496b-6xtld\" (UID: \"418e7a93-3e5c-41d8-a0be-bf74b3919943\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.023950 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7e7e895-264d-4057-a3cc-602abc783f04-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024034 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-multus-certs\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024060 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-slash\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024088 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e60a880f-7773-41dd-9995-3b1719fe8878-data-volume\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024114 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d7e7e895-264d-4057-a3cc-602abc783f04-signing-cabundle\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024129 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99f5v\" (UniqueName: \"kubernetes.io/projected/d409cf23-c109-4dfb-a569-5041531269dc-kube-api-access-99f5v\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024156 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-dbus\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024173 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e60a880f-7773-41dd-9995-3b1719fe8878-data-volume\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024182 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-multus\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024222 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjsf\" (UniqueName: \"kubernetes.io/projected/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-kube-api-access-zxjsf\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024250 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024275 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-systemd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024298 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-ovn\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024343 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.024374 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024371 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024410 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-bin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024433 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-hostroot\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024479 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024508 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c751f98-9133-4738-9052-8dba6020cda6-config-volume\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024549 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024650 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7e7e895-264d-4057-a3cc-602abc783f04-signing-key\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024690 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc8z\" (UniqueName: \"kubernetes.io/projected/9c7c8096-81d8-4c14-b102-08569d25da31-kube-api-access-2pc8z\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024721 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-snapshots\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024745 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbjm\" (UniqueName: \"kubernetes.io/projected/e60a880f-7773-41dd-9995-3b1719fe8878-kube-api-access-fzbjm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024750 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c751f98-9133-4738-9052-8dba6020cda6-config-volume\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024804 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0554ee01-1433-4bd7-9a51-d74c757af46a-agent-certs\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024808 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cea9a4e-e914-40b6-9237-345be4aff28c-original-pull-secret\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024831 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-config\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024845 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d7e7e895-264d-4057-a3cc-602abc783f04-signing-key\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024860 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-env-overrides\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024886 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024834 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bcf0cc-7eef-4cbe-9d72-140932580c56-snapshots\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024911 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024936 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024941 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.024995 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/698d7a34-d950-442b-b12e-b0954c8fe873-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025019 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-ovnkube-config\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025024 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdgb7\" (UniqueName: \"kubernetes.io/projected/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9-kube-api-access-fdgb7\") pod \"migrator-66df5d496b-h4gtf\" (UID: \"ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025042 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0554ee01-1433-4bd7-9a51-d74c757af46a-agent-certs\") pod \"konnectivity-agent-b4cc4\" (UID: \"0554ee01-1433-4bd7-9a51-d74c757af46a\") " pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025074 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a8a0aad-3971-455a-8fda-f40c28d8bc13-env-overrides\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.025226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025103 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.025951 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025271 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/698d7a34-d950-442b-b12e-b0954c8fe873-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:47.025951 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025322 5074 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 12:31:47.025951 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.025422 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.027487 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.027469 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/746d043e-1f60-464f-ba50-6a437eb9089f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sckc2\" (UID: \"746d043e-1f60-464f-ba50-6a437eb9089f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:47.046083 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.046063 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmcf\" (UniqueName: \"kubernetes.io/projected/579b4046-76c9-4d7e-b77a-9888510a22a8-kube-api-access-8pmcf\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.049741 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.049726 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.051455 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.051439 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5nm4\" (UniqueName: \"kubernetes.io/projected/bf409f2b-9f95-4f84-bbd6-d9cc841fc074-kube-api-access-j5nm4\") pod \"network-check-source-6f78d59f99-62jh5\" (UID: \"bf409f2b-9f95-4f84-bbd6-d9cc841fc074\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62jh5" Oct 31 12:31:47.052270 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.052254 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r6x\" (UniqueName: \"kubernetes.io/projected/55bcf0cc-7eef-4cbe-9d72-140932580c56-kube-api-access-p9r6x\") pod \"insights-operator-7c4986d449-tc2tp\" (UID: \"55bcf0cc-7eef-4cbe-9d72-140932580c56\") " pod="openshift-insights/insights-operator-7c4986d449-tc2tp" Oct 31 12:31:47.054313 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.054255 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7tbp\" (UniqueName: \"kubernetes.io/projected/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-kube-api-access-n7tbp\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:47.055024 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.055008 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4sk5\" (UniqueName: \"kubernetes.io/projected/f05d824e-3be1-46f6-8df0-2ade9675e534-kube-api-access-c4sk5\") pod \"network-check-target-gjvxs\" (UID: \"f05d824e-3be1-46f6-8df0-2ade9675e534\") " pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:47.056413 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.056397 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqtb\" (UniqueName: \"kubernetes.io/projected/a069e898-8264-4573-b885-2528dadb0305-kube-api-access-6pqtb\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:47.061152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.061135 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4h8n\" (UniqueName: \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\") pod \"periodic-gathering-7vhj5-f2fsm\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:31:47.061697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.061682 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxc57\" (UniqueName: \"kubernetes.io/projected/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kube-api-access-wxc57\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.061905 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.061888 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgq6\" (UniqueName: \"kubernetes.io/projected/8c751f98-9133-4738-9052-8dba6020cda6-kube-api-access-skgq6\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.062674 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.062659 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv66w\" (UniqueName: \"kubernetes.io/projected/4a8a0aad-3971-455a-8fda-f40c28d8bc13-kube-api-access-fv66w\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.063206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.063192 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.065247 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.065227 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-757tm\" (UniqueName: \"kubernetes.io/projected/59030007-53c1-42c9-baec-a4d2ec90f3dd-kube-api-access-757tm\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.065299 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.065262 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.070066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.070050 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbjm\" (UniqueName: \"kubernetes.io/projected/e60a880f-7773-41dd-9995-3b1719fe8878-kube-api-access-fzbjm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.074779 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.074763 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8rb\" (UniqueName: \"kubernetes.io/projected/418e7a93-3e5c-41d8-a0be-bf74b3919943-kube-api-access-9l8rb\") pod \"migrator-66df5d496b-6xtld\" (UID: \"418e7a93-3e5c-41d8-a0be-bf74b3919943\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6xtld" Oct 31 12:31:47.074967 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.074947 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrdr\" (UniqueName: \"kubernetes.io/projected/fee2a93c-fe10-47e5-840c-8d8be03b2715-kube-api-access-zqrdr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8\" (UID: \"fee2a93c-fe10-47e5-840c-8d8be03b2715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" Oct 31 12:31:47.075017 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.074948 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc8z\" (UniqueName: \"kubernetes.io/projected/9c7c8096-81d8-4c14-b102-08569d25da31-kube-api-access-2pc8z\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.075493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.075479 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjsf\" (UniqueName: \"kubernetes.io/projected/e040a356-8196-40c6-8f2c-a94bb4c6bfe4-kube-api-access-zxjsf\") pod \"ingress-canary-bqcvc\" (UID: \"e040a356-8196-40c6-8f2c-a94bb4c6bfe4\") " pod="openshift-ingress-canary/ingress-canary-bqcvc" Oct 31 12:31:47.075684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.075665 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntcw7\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.075761 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.075747 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhdx\" (UniqueName: \"kubernetes.io/projected/fb0b76f8-eb8d-45ee-afec-b393223b0504-kube-api-access-wzhdx\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:47.077798 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.077781 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd54w\" (UniqueName: \"kubernetes.io/projected/d7e7e895-264d-4057-a3cc-602abc783f04-kube-api-access-qd54w\") pod \"service-ca-5fffc97bdf-sjz88\" (UID: \"d7e7e895-264d-4057-a3cc-602abc783f04\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sjz88" Oct 31 12:31:47.077892 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.077876 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdgb7\" (UniqueName: \"kubernetes.io/projected/ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9-kube-api-access-fdgb7\") pod \"migrator-66df5d496b-h4gtf\" (UID: \"ec5b413d-7afc-4d3d-ae8f-12c9ce8439e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h4gtf" Oct 31 12:31:47.080039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.080023 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/f0f7cd28-a6f8-43af-b1dc-f85855564643-kube-api-access-sxhlm\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.080090 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.080047 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99f5v\" (UniqueName: \"kubernetes.io/projected/d409cf23-c109-4dfb-a569-5041531269dc-kube-api-access-99f5v\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:47.125542 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125508 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.125672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125544 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.125672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125567 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-node-log\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.125672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125588 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-kubelet\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.125672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125613 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.125672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125613 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.125672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125652 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-node-log\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.125672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125662 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-kubelet\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.125671 5074 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125691 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125659 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-modprobe-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.125738 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.625723847 +0000 UTC m=+1.803441816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125762 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-etc-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125767 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-modprobe-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125784 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-log-socket\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125810 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a069e898-8264-4573-b885-2528dadb0305-hosts-file\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125831 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-log-socket\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125833 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-host\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125811 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-etc-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125864 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bcc043c-d8ac-4f2a-ae8d-3e233db2842d-host\") pod \"node-ca-qkx64\" (UID: \"0bcc043c-d8ac-4f2a-ae8d-3e233db2842d\") " pod="openshift-image-registry/node-ca-qkx64" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125860 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a069e898-8264-4573-b885-2528dadb0305-hosts-file\") pod \"node-resolver-9v4bh\" (UID: \"a069e898-8264-4573-b885-2528dadb0305\") " pod="openshift-dns/node-resolver-9v4bh" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125878 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-conf\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125901 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-run\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125925 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-cnibin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125941 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-etc-kubernetes\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125971 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-run\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.125990 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-cnibin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126029 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-etc-kubernetes\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126028 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-conf\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126049 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-netns\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126070 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-netns\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126071 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126107 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126109 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-conf-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126142 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-sys-fs\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126146 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-conf-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126169 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-registration-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126194 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-sys-fs\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126194 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-host\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126239 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-system-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126245 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-registration-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126265 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-socket-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126286 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-host\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126294 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-system-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126289 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-device-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126337 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-socket-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126338 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-device-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126337 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126374 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e60a880f-7773-41dd-9995-3b1719fe8878-crio-socket\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126390 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126403 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-lib-modules\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.126403 5074 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126451 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-multus-certs\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.126456 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.626442577 +0000 UTC m=+1.804160540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126463 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e60a880f-7773-41dd-9995-3b1719fe8878-crio-socket\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126486 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-multus-certs\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126478 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-slash\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126522 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-slash\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126512 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-lib-modules\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126520 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-dbus\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126564 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-dbus\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126562 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-multus\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126596 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-systemd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126596 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-multus\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126621 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-ovn\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126664 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-systemd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126690 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126713 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-ovn\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126758 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126785 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-bin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.126786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126806 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-hostroot\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.126831 5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126830 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126861 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-var-lib-cni-bin\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126888 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-hostroot\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.126871 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.626858637 +0000 UTC m=+1.804576604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126864 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-run-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126967 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126982 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e60a880f-7773-41dd-9995-3b1719fe8878-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.126993 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-system-cni-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127021 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409cf23-c109-4dfb-a569-5041531269dc-host-slash\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127038 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-systemd\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127053 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-netd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127075 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-system-cni-dir\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127082 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysctl-d\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127095 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-kubernetes\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127092 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d409cf23-c109-4dfb-a569-5041531269dc-host-slash\") pod \"iptables-alerter-6hdft\" (UID: \"d409cf23-c109-4dfb-a569-5041531269dc\") " pod="openshift-network-operator/iptables-alerter-6hdft" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127104 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-systemd\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127122 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-k8s-cni-cncf-io\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127138 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-netd\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127143 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-host-run-k8s-cni-cncf-io\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127138 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-kubernetes\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127154 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127182 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127207 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-systemd-units\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127230 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127259 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-kubelet-config\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127265 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-systemd-units\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.127280 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.627266446 +0000 UTC m=+1.804984447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127303 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cea9a4e-e914-40b6-9237-345be4aff28c-kubelet-config\") pod \"global-pull-secret-syncer-sk7rm\" (UID: \"8cea9a4e-e914-40b6-9237-345be4aff28c\") " pod="kube-system/global-pull-secret-syncer-sk7rm" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127311 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysconfig\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127346 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.127355 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.127368 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127370 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127370 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-etc-sysconfig\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127412 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.127418 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.627408647 +0000 UTC m=+1.805126644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127442 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-kubelet\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.127436 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.127505 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127522 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-var-lib-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.127538 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:47.627526858 +0000 UTC m=+1.805244833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127460 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-kubelet\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127491 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-var-lib-openvswitch\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127582 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-sys\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127608 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127654 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-cnibin\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127668 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-sys\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127678 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-netns\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127719 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-cni-dir\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127735 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59030007-53c1-42c9-baec-a4d2ec90f3dd-cnibin\") pod \"multus-additional-cni-plugins-9tkzk\" (UID: \"59030007-53c1-42c9-baec-a4d2ec90f3dd\") " pod="openshift-multus/multus-additional-cni-plugins-9tkzk" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127770 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-run-netns\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127799 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-var-lib-kubelet\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127831 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-socket-dir-parent\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127857 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127867 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0f7cd28-a6f8-43af-b1dc-f85855564643-var-lib-kubelet\") pod \"tuned-7z7sx\" (UID: \"f0f7cd28-a6f8-43af-b1dc-f85855564643\") " pod="openshift-cluster-node-tuning-operator/tuned-7z7sx" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127881 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-bin\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127898 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c7c8096-81d8-4c14-b102-08569d25da31-multus-socket-dir-parent\") pod \"multus-db7cd\" (UID: \"9c7c8096-81d8-4c14-b102-08569d25da31\") " pod="openshift-multus/multus-db7cd" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127920 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f23f959-82b3-48cc-8f17-917d8c8c6f6b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwmz6\" (UID: \"2f23f959-82b3-48cc-8f17-917d8c8c6f6b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwmz6" Oct 31 12:31:47.128297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.127970 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a8a0aad-3971-455a-8fda-f40c28d8bc13-host-cni-bin\") pod \"ovnkube-node-r8m9k\" (UID: \"4a8a0aad-3971-455a-8fda-f40c28d8bc13\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:47.130039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.128991 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c751f98-9133-4738-9052-8dba6020cda6-metrics-tls\") pod \"dns-default-p4psl\" (UID: \"8c751f98-9133-4738-9052-8dba6020cda6\") " pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.130039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.129227 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0b76f8-eb8d-45ee-afec-b393223b0504-metrics-certs\") pod \"network-metrics-daemon-5kl4n\" (UID: \"fb0b76f8-eb8d-45ee-afec-b393223b0504\") " pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:47.199416 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.199113 5074 scope.go:122] "RemoveContainer" containerID="67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692" Oct 31 12:31:47.199677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.199624 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" Oct 31 12:31:47.208846 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.208826 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kl4n" Oct 31 12:31:47.215391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.215370 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:47.481304 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.481276 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2"] Oct 31 12:31:47.511015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.510988 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4psl"] Oct 31 12:31:47.514216 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:47.514185 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c751f98_9133_4738_9052_8dba6020cda6.slice/crio-46dee80290839647dee620e7d7a455840056d64e5f7e7cf1afa59b23734e1ba5 WatchSource:0}: Error finding container 46dee80290839647dee620e7d7a455840056d64e5f7e7cf1afa59b23734e1ba5: Status 404 returned error can't find the container with id 46dee80290839647dee620e7d7a455840056d64e5f7e7cf1afa59b23734e1ba5 Oct 31 12:31:47.515532 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.515514 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:31:47.518623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.518552 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kl4n"] Oct 31 12:31:47.520741 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:47.520721 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0b76f8_eb8d_45ee_afec_b393223b0504.slice/crio-be0eb9497c2a282bd449212b2c243d53e6135cc647e94b0724e619fc25162051 WatchSource:0}: Error finding container be0eb9497c2a282bd449212b2c243d53e6135cc647e94b0724e619fc25162051: Status 404 returned error can't find the container with id be0eb9497c2a282bd449212b2c243d53e6135cc647e94b0724e619fc25162051 Oct 31 12:31:47.533177 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.533159 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:47.533304 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.533288 5074 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:47.533350 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.533341 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:48.533326709 +0000 UTC m=+2.711044687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:47.633769 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.633737 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:47.633954 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.633782 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:47.633954 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.633912 5074 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:47.634044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.633919 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.634044 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.633984 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:48.633966772 +0000 UTC m=+2.811684744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:47.634044 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634006 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:48.633998143 +0000 UTC m=+2.811716105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:47.634044 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634020 5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:47.634044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.634029 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634061 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:48.634049985 +0000 UTC m=+2.811767947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.634076 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634092 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634102 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:47.634124 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634135 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:48.634122613 +0000 UTC m=+2.811840576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634181 5074 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634199 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634215 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:47.634231 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634221 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:48.634208909 +0000 UTC m=+2.811926873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:47.634564 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:47.634248 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:48.634239161 +0000 UTC m=+2.811957137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:48.006739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.006618 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" event={"ID":"746d043e-1f60-464f-ba50-6a437eb9089f","Type":"ContainerStarted","Data":"8f4ec65514335cc0ec086cc4589eeb0a809d449f4145d89f3b5602d4f7d1a94b"} Oct 31 12:31:48.009914 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.009873 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:31:48.010411 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.010389 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/1.log" Oct 31 12:31:48.010529 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.010433 5074 generic.go:356] "Generic (PLEG): container finished" podID="a8053559-57b0-468c-a369-893764b0ed4d" containerID="95f04d7899f5f4130c7b6e9f274ea04e7e15edc64a6b2467eb19606d4421a921" exitCode=255 Oct 31 12:31:48.010588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.010530 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" event={"ID":"a8053559-57b0-468c-a369-893764b0ed4d","Type":"ContainerDied","Data":"95f04d7899f5f4130c7b6e9f274ea04e7e15edc64a6b2467eb19606d4421a921"} Oct 31 12:31:48.010588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.010575 5074 scope.go:122] "RemoveContainer" containerID="67a622b1bf222e2fc66d786ae089f07e6f97253f3eab1d8c0206b76006bb3692" Oct 31 12:31:48.011095 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.011042 5074 scope.go:122] "RemoveContainer" containerID="95f04d7899f5f4130c7b6e9f274ea04e7e15edc64a6b2467eb19606d4421a921" Oct 31 12:31:48.011373 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.011267 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vnxg_openshift-console-operator(a8053559-57b0-468c-a369-893764b0ed4d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" podUID="a8053559-57b0-468c-a369-893764b0ed4d" Oct 31 12:31:48.012724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.012665 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4psl" event={"ID":"8c751f98-9133-4738-9052-8dba6020cda6","Type":"ContainerStarted","Data":"46dee80290839647dee620e7d7a455840056d64e5f7e7cf1afa59b23734e1ba5"} Oct 31 12:31:48.014341 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.014320 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kl4n" event={"ID":"fb0b76f8-eb8d-45ee-afec-b393223b0504","Type":"ContainerStarted","Data":"be0eb9497c2a282bd449212b2c243d53e6135cc647e94b0724e619fc25162051"} Oct 31 12:31:48.544876 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.544838 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:48.545064 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.544993 5074 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:48.545064 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.545062 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls podName:dee0dda1-07c9-43fc-b7d6-f09b502d0149 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.545043929 +0000 UTC m=+4.722761891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vzqxc" (UID: "dee0dda1-07c9-43fc-b7d6-f09b502d0149") : secret "cluster-monitoring-operator-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.645877 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.645949 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.646003 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646022 5074 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.646031 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.646077 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:48.646126 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646262 5074 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646304 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.646289096 +0000 UTC m=+4.824007072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646362 5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646395 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.646383949 +0000 UTC m=+4.824101925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646466 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.646454798 +0000 UTC m=+4.824172774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646468 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646482 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646500 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646510 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646511 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.646502369 +0000 UTC m=+4.824220348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646548 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.64653843 +0000 UTC m=+4.824256396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:48.646577 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:48.646561 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:50.646553702 +0000 UTC m=+4.824271668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:49.010288 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:49.010249 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:31:49.034088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:49.034057 5074 generic.go:356] "Generic (PLEG): container finished" podID="55bcf0cc-7eef-4cbe-9d72-140932580c56" containerID="ca27f2953fe244c854deb997a36624a4104c309f305f79f145f614f242e1bfb0" exitCode=0 Oct 31 12:31:49.034221 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:49.034150 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" event={"ID":"55bcf0cc-7eef-4cbe-9d72-140932580c56","Type":"ContainerDied","Data":"ca27f2953fe244c854deb997a36624a4104c309f305f79f145f614f242e1bfb0"} Oct 31 12:31:49.034513 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:49.034496 5074 scope.go:122] "RemoveContainer" containerID="ca27f2953fe244c854deb997a36624a4104c309f305f79f145f614f242e1bfb0" Oct 31 12:31:49.036893 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:49.036877 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:31:49.042544 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:49.042516 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kl4n" event={"ID":"fb0b76f8-eb8d-45ee-afec-b393223b0504","Type":"ContainerStarted","Data":"838d0da28d0e0e8dab57d199eccbc40fb97e43909958f287d5fcf77f9c01c674"} Oct 31 12:31:49.042699 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:49.042572 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kl4n" event={"ID":"fb0b76f8-eb8d-45ee-afec-b393223b0504","Type":"ContainerStarted","Data":"3d0c29eba00c15548bc03886fec92cd3de8eeb084a6dfe68378d6f5ae06b7176"} Oct 31 12:31:50.048982 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.048940 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4psl" event={"ID":"8c751f98-9133-4738-9052-8dba6020cda6","Type":"ContainerStarted","Data":"38af9289b9737c84fb728dbf923c0755cd6e1a8a6bc0eac32116b2f16879be8c"} Oct 31 12:31:50.049402 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.048999 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4psl" event={"ID":"8c751f98-9133-4738-9052-8dba6020cda6","Type":"ContainerStarted","Data":"e2fc9d6827d44bc839f2143c3d0186fe231e3c9d0dd9f0fc62198531b8239617"} Oct 31 12:31:50.049402 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.049029 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p4psl" Oct 31 12:31:50.050843 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.050814 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" event={"ID":"746d043e-1f60-464f-ba50-6a437eb9089f","Type":"ContainerStarted","Data":"0d4254c8f92a319e648099fa6441c1870aa1032068e0a404c93518e722a55c3f"} Oct 31 12:31:50.051032 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.050871 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sckc2" event={"ID":"746d043e-1f60-464f-ba50-6a437eb9089f","Type":"ContainerStarted","Data":"c42adc527f2f392e9cf2f9a64144bd14f4a95f23b8871739b1a434aae646b01d"} Oct 31 12:31:50.052762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.052735 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" event={"ID":"55bcf0cc-7eef-4cbe-9d72-140932580c56","Type":"ContainerStarted","Data":"47548d4aae7f66ea223e0af10d62a43aeb4275befc01060fcc03b41f2096d775"} Oct 31 12:31:50.565190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.565152 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:50.567568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.567540 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dee0dda1-07c9-43fc-b7d6-f09b502d0149-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vzqxc\" (UID: \"dee0dda1-07c9-43fc-b7d6-f09b502d0149\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:50.665823 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.665790 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:50.665996 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.665834 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:50.665996 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.665866 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:50.665996 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.665886 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:50.665996 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.665961 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:50.665996 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.665974 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dd98c8f4-j6vtq: secret "image-registry-tls" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666010 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls podName:300d5937-06ea-46c4-894e-17ab3aea7815 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:54.665997949 +0000 UTC m=+8.843715932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls") pod "image-registry-85dd98c8f4-j6vtq" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815") : secret "image-registry-tls" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.666004 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.665959 5074 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666048 5074 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666056 5074 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8479bcff-g6cvb: secret "image-registry-tls" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666065 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:54.666045259 +0000 UTC m=+8.843763225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : configmap references non-existent config key: service-ca.crt Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.665959 5074 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666098 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls podName:a4942177-472e-47cf-934b-92b58447dfae nodeName:}" failed. No retries permitted until 2026-10-31 12:31:54.666084645 +0000 UTC m=+8.843802607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls") pod "image-registry-6f8479bcff-g6cvb" (UID: "a4942177-472e-47cf-934b-92b58447dfae") : secret "image-registry-tls" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666117 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls podName:e60a880f-7773-41dd-9995-3b1719fe8878 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:54.666108309 +0000 UTC m=+8.843826278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc7gp" (UID: "e60a880f-7773-41dd-9995-3b1719fe8878") : secret "insights-runtime-extractor-tls" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666136 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert podName:698d7a34-d950-442b-b12e-b0954c8fe873 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:54.666127727 +0000 UTC m=+8.843845689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pnzjp" (UID: "698d7a34-d950-442b-b12e-b0954c8fe873") : secret "networking-console-plugin-cert" not found Oct 31 12:31:50.666255 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.666173 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:50.666619 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666276 5074 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 12:31:50.666619 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:50.666326 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs podName:579b4046-76c9-4d7e-b77a-9888510a22a8 nodeName:}" failed. No retries permitted until 2026-10-31 12:31:54.666314494 +0000 UTC m=+8.844032470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs") pod "router-default-865bf4d8bc-d5gwq" (UID: "579b4046-76c9-4d7e-b77a-9888510a22a8") : secret "router-metrics-certs-default" not found Oct 31 12:31:50.799259 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:50.799233 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" Oct 31 12:31:51.057620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:51.057592 5074 generic.go:356] "Generic (PLEG): container finished" podID="fee2a93c-fe10-47e5-840c-8d8be03b2715" containerID="061fed8e866a4a43db916f9cede7be91c5c27c3261b69c8142b9121ebd46ba07" exitCode=0 Oct 31 12:31:51.058149 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:51.058123 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" event={"ID":"fee2a93c-fe10-47e5-840c-8d8be03b2715","Type":"ContainerDied","Data":"061fed8e866a4a43db916f9cede7be91c5c27c3261b69c8142b9121ebd46ba07"} Oct 31 12:31:51.058449 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:51.058433 5074 scope.go:122] "RemoveContainer" containerID="061fed8e866a4a43db916f9cede7be91c5c27c3261b69c8142b9121ebd46ba07" Oct 31 12:31:51.081890 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:51.080688 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc"] Oct 31 12:31:52.063415 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:52.063369 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nm7d8" event={"ID":"fee2a93c-fe10-47e5-840c-8d8be03b2715","Type":"ContainerStarted","Data":"7d42e4943645ac0a1d0e891bf5e579576dc7c3847b266ef381781624f6d564ef"} Oct 31 12:31:52.065033 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:52.065005 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" event={"ID":"dee0dda1-07c9-43fc-b7d6-f09b502d0149","Type":"ContainerStarted","Data":"9de7575ff7e2c8b7b7f3aec41d870eee6b43937d1c28467562d0e1c59628e0e7"} Oct 31 12:31:53.346395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:53.346369 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:53.382137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:53.382101 5074 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:53.382490 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:53.382471 5074 scope.go:122] "RemoveContainer" containerID="95f04d7899f5f4130c7b6e9f274ea04e7e15edc64a6b2467eb19606d4421a921" Oct 31 12:31:53.382740 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:53.382718 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vnxg_openshift-console-operator(a8053559-57b0-468c-a369-893764b0ed4d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" podUID="a8053559-57b0-468c-a369-893764b0ed4d" Oct 31 12:31:53.727505 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:53.727477 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b4cc4" Oct 31 12:31:54.074287 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.074198 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vzqxc" event={"ID":"dee0dda1-07c9-43fc-b7d6-f09b502d0149","Type":"ContainerStarted","Data":"5145ca44fb4500bd66a3d2328fdc73325b63061f57767b5bc09070163459bd39"} Oct 31 12:31:54.407610 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.407525 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:54.408013 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.407911 5074 scope.go:122] "RemoveContainer" containerID="95f04d7899f5f4130c7b6e9f274ea04e7e15edc64a6b2467eb19606d4421a921" Oct 31 12:31:54.408100 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:54.408079 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vnxg_openshift-console-operator(a8053559-57b0-468c-a369-893764b0ed4d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" podUID="a8053559-57b0-468c-a369-893764b0ed4d" Oct 31 12:31:54.704224 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.704138 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:54.704224 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.704186 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:54.704224 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.704219 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:54.704494 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.704246 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:54.704494 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.704268 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:54.704494 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.704298 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:54.705117 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.705082 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/579b4046-76c9-4d7e-b77a-9888510a22a8-service-ca-bundle\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:54.706858 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.706824 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"image-registry-6f8479bcff-g6cvb\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:54.706858 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.706851 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/698d7a34-d950-442b-b12e-b0954c8fe873-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pnzjp\" (UID: \"698d7a34-d950-442b-b12e-b0954c8fe873\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:54.707053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.706869 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579b4046-76c9-4d7e-b77a-9888510a22a8-metrics-certs\") pod \"router-default-865bf4d8bc-d5gwq\" (UID: \"579b4046-76c9-4d7e-b77a-9888510a22a8\") " pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:54.707053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.707012 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e60a880f-7773-41dd-9995-3b1719fe8878-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc7gp\" (UID: \"e60a880f-7773-41dd-9995-3b1719fe8878\") " pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:54.707168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.707151 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"image-registry-85dd98c8f4-j6vtq\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:54.713960 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.713945 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:31:54.716681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:54.716661 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jc7gp" Oct 31 12:31:55.001224 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.001189 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:55.002346 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.002312 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" Oct 31 12:31:55.002346 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.002348 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:55.003686 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.003598 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8479bcff-g6cvb"] Oct 31 12:31:55.004896 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.004854 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jc7gp"] Oct 31 12:31:55.008148 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:55.008127 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4942177_472e_47cf_934b_92b58447dfae.slice/crio-dc1b0ef4305e4b51daf6d10e35406e186e94d6522f2da274fd7268f84d8a121b WatchSource:0}: Error finding container dc1b0ef4305e4b51daf6d10e35406e186e94d6522f2da274fd7268f84d8a121b: Status 404 returned error can't find the container with id dc1b0ef4305e4b51daf6d10e35406e186e94d6522f2da274fd7268f84d8a121b Oct 31 12:31:55.011470 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:55.011447 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60a880f_7773_41dd_9995_3b1719fe8878.slice/crio-2595863d97a799073f204a587cec076eef42e1a16fcecd22b07fc15d8d792bdc WatchSource:0}: Error finding container 2595863d97a799073f204a587cec076eef42e1a16fcecd22b07fc15d8d792bdc: Status 404 returned error can't find the container with id 2595863d97a799073f204a587cec076eef42e1a16fcecd22b07fc15d8d792bdc Oct 31 12:31:55.096052 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.096014 5074 generic.go:356] "Generic (PLEG): container finished" podID="55bcf0cc-7eef-4cbe-9d72-140932580c56" containerID="47548d4aae7f66ea223e0af10d62a43aeb4275befc01060fcc03b41f2096d775" exitCode=0 Oct 31 12:31:55.096173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.096094 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" event={"ID":"55bcf0cc-7eef-4cbe-9d72-140932580c56","Type":"ContainerDied","Data":"47548d4aae7f66ea223e0af10d62a43aeb4275befc01060fcc03b41f2096d775"} Oct 31 12:31:55.096173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.096168 5074 scope.go:122] "RemoveContainer" containerID="ca27f2953fe244c854deb997a36624a4104c309f305f79f145f614f242e1bfb0" Oct 31 12:31:55.096551 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.096531 5074 scope.go:122] "RemoveContainer" containerID="47548d4aae7f66ea223e0af10d62a43aeb4275befc01060fcc03b41f2096d775" Oct 31 12:31:55.096811 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:31:55.096787 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-tc2tp_openshift-insights(55bcf0cc-7eef-4cbe-9d72-140932580c56)\"" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" podUID="55bcf0cc-7eef-4cbe-9d72-140932580c56" Oct 31 12:31:55.104300 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.104213 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc7gp" event={"ID":"e60a880f-7773-41dd-9995-3b1719fe8878","Type":"ContainerStarted","Data":"2595863d97a799073f204a587cec076eef42e1a16fcecd22b07fc15d8d792bdc"} Oct 31 12:31:55.108940 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.107655 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" event={"ID":"a4942177-472e-47cf-934b-92b58447dfae","Type":"ContainerStarted","Data":"dc1b0ef4305e4b51daf6d10e35406e186e94d6522f2da274fd7268f84d8a121b"} Oct 31 12:31:55.299956 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.299867 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-865bf4d8bc-d5gwq"] Oct 31 12:31:55.303906 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.303880 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85dd98c8f4-j6vtq"] Oct 31 12:31:55.306390 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:55.306362 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pnzjp"] Oct 31 12:31:55.306509 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:55.306455 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579b4046_76c9_4d7e_b77a_9888510a22a8.slice/crio-25993f8bb8280a974e99ef68ff6291d81e310853493a900eafc8d0402195627a WatchSource:0}: Error finding container 25993f8bb8280a974e99ef68ff6291d81e310853493a900eafc8d0402195627a: Status 404 returned error can't find the container with id 25993f8bb8280a974e99ef68ff6291d81e310853493a900eafc8d0402195627a Oct 31 12:31:55.310857 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:31:55.310835 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698d7a34_d950_442b_b12e_b0954c8fe873.slice/crio-688eacc2188362725fdae446007cbd6a1eeda5e5ae19306b69a3368eaece085d WatchSource:0}: Error finding container 688eacc2188362725fdae446007cbd6a1eeda5e5ae19306b69a3368eaece085d: Status 404 returned error can't find the container with id 688eacc2188362725fdae446007cbd6a1eeda5e5ae19306b69a3368eaece085d Oct 31 12:31:56.114531 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.114458 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" event={"ID":"579b4046-76c9-4d7e-b77a-9888510a22a8","Type":"ContainerStarted","Data":"5770ab1c4c5241ab948828c4cf7a7856b665bfcf6f9934d1c4404dace30b1f74"} Oct 31 12:31:56.114899 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.114534 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" event={"ID":"579b4046-76c9-4d7e-b77a-9888510a22a8","Type":"ContainerStarted","Data":"25993f8bb8280a974e99ef68ff6291d81e310853493a900eafc8d0402195627a"} Oct 31 12:31:56.117150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.117119 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc7gp" event={"ID":"e60a880f-7773-41dd-9995-3b1719fe8878","Type":"ContainerStarted","Data":"d1bbfe461291fba82b785fd1816ad9426fc5935efd14e8f30790f9f87dc22f62"} Oct 31 12:31:56.117274 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.117180 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc7gp" event={"ID":"e60a880f-7773-41dd-9995-3b1719fe8878","Type":"ContainerStarted","Data":"975eda0600d67531b25985acb9f1b01249af43fba1b013b936787d281df75900"} Oct 31 12:31:56.119362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.119342 5074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:31:56.119464 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.119365 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" event={"ID":"a4942177-472e-47cf-934b-92b58447dfae","Type":"ContainerStarted","Data":"2a303f85cb3b918bd74447013462bfa15c89eec8acab327d0b5cf98b6e8c49d6"} Oct 31 12:31:56.122183 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.122139 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" event={"ID":"300d5937-06ea-46c4-894e-17ab3aea7815","Type":"ContainerStarted","Data":"611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b"} Oct 31 12:31:56.122183 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.122182 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" event={"ID":"300d5937-06ea-46c4-894e-17ab3aea7815","Type":"ContainerStarted","Data":"5f905b7c80a505b7a0cc1332df363d8f2efb440a75bd40036104d71dd514a7c2"} Oct 31 12:31:56.123545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:56.123521 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" event={"ID":"698d7a34-d950-442b-b12e-b0954c8fe873","Type":"ContainerStarted","Data":"688eacc2188362725fdae446007cbd6a1eeda5e5ae19306b69a3368eaece085d"} Oct 31 12:31:57.001957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.001918 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:57.004809 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.004774 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:57.116910 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.116880 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:31:57.117321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.117091 5074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:31:57.130878 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.130798 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pnzjp" event={"ID":"698d7a34-d950-442b-b12e-b0954c8fe873","Type":"ContainerStarted","Data":"091bf7fc171513cf1a7a932df1fc0868ef589abd8a8d984937d22237d5677119"} Oct 31 12:31:57.130878 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.130831 5074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:31:57.131108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.130953 5074 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 12:31:57.258816 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.258741 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:31:57.467207 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.467166 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:57.468689 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:57.468663 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-865bf4d8bc-d5gwq" Oct 31 12:31:58.136365 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:58.136264 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc7gp" event={"ID":"e60a880f-7773-41dd-9995-3b1719fe8878","Type":"ContainerStarted","Data":"83c62f806233f4e8241fde8e6185209a87425f86fffba9f692faf6605e17c057"} Oct 31 12:31:58.930929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:58.930897 5074 scope.go:122] "RemoveContainer" containerID="95f04d7899f5f4130c7b6e9f274ea04e7e15edc64a6b2467eb19606d4421a921" Oct 31 12:31:59.142797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:59.142765 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:31:59.143268 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:59.142882 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" event={"ID":"a8053559-57b0-468c-a369-893764b0ed4d","Type":"ContainerStarted","Data":"c8046c780bf3c5457c2fd4497b115204e240f22ff822494c4c734597845945fe"} Oct 31 12:31:59.143455 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:59.143430 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:31:59.369009 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:31:59.368977 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vnxg" Oct 31 12:32:00.060139 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:00.060112 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p4psl" Oct 31 12:32:00.464703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:00.464599 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:32:04.931154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:04.931121 5074 scope.go:122] "RemoveContainer" containerID="47548d4aae7f66ea223e0af10d62a43aeb4275befc01060fcc03b41f2096d775" Oct 31 12:32:05.169208 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.169172 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tc2tp" event={"ID":"55bcf0cc-7eef-4cbe-9d72-140932580c56","Type":"ContainerStarted","Data":"243913f16aa3875df99e172b94dc04371d36fceca941a048c4d9623b4b6b25e6"} Oct 31 12:32:05.430360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.430324 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t6v9c-694pl"] Oct 31 12:32:05.434328 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.434299 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.454305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.453487 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t6v9c-694pl"] Oct 31 12:32:05.463438 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.463384 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85dd98c8f4-j6vtq"] Oct 31 12:32:05.465719 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.465676 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p"] Oct 31 12:32:05.470045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.470020 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-fmzxl"] Oct 31 12:32:05.470181 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.470158 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" Oct 31 12:32:05.474310 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.474286 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.482506 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.482478 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p"] Oct 31 12:32:05.491949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.489019 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q82lv\"" Oct 31 12:32:05.491949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.489108 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 12:32:05.491949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.489267 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fmzxl"] Oct 31 12:32:05.491949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.491516 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 12:32:05.491949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.491574 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 12:32:05.558066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.558032 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cb45b6b6f-8pjrj"] Oct 31 12:32:05.561454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.561429 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.577926 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.577885 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb45b6b6f-8pjrj"] Oct 31 12:32:05.585695 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.585667 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5rxxb\"" Oct 31 12:32:05.597095 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.597060 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv6fv\" (UniqueName: \"kubernetes.io/projected/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-kube-api-access-lv6fv\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.597298 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.597157 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9551cad5-7a59-43fe-af7a-ff0ed70a8789-tmp\") pod \"downloads-6dd789c8b7-fmzxl\" (UID: \"9551cad5-7a59-43fe-af7a-ff0ed70a8789\") " pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.597298 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.597218 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-service-ca-bundle\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.597658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.597324 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/486b788d-7531-41c1-89d5-996d6a05c0d9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7dq5p\" (UID: \"486b788d-7531-41c1-89d5-996d6a05c0d9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" Oct 31 12:32:05.597658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.597392 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99l8\" (UniqueName: \"kubernetes.io/projected/9551cad5-7a59-43fe-af7a-ff0ed70a8789-kube-api-access-z99l8\") pod \"downloads-6dd789c8b7-fmzxl\" (UID: \"9551cad5-7a59-43fe-af7a-ff0ed70a8789\") " pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.597658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.597473 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-archives-path\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.597658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.597513 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-trusted-ca-bundle\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.698317 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698218 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv6fv\" (UniqueName: \"kubernetes.io/projected/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-kube-api-access-lv6fv\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.698317 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698298 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9551cad5-7a59-43fe-af7a-ff0ed70a8789-tmp\") pod \"downloads-6dd789c8b7-fmzxl\" (UID: \"9551cad5-7a59-43fe-af7a-ff0ed70a8789\") " pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698325 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-service-ca-bundle\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698353 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/486b788d-7531-41c1-89d5-996d6a05c0d9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7dq5p\" (UID: \"486b788d-7531-41c1-89d5-996d6a05c0d9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698383 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/948aab57-f974-48a1-b43a-cb75577e8d7a-ca-trust-extracted\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698409 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-bound-sa-token\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698434 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99l8\" (UniqueName: \"kubernetes.io/projected/9551cad5-7a59-43fe-af7a-ff0ed70a8789-kube-api-access-z99l8\") pod \"downloads-6dd789c8b7-fmzxl\" (UID: \"9551cad5-7a59-43fe-af7a-ff0ed70a8789\") " pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698467 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/948aab57-f974-48a1-b43a-cb75577e8d7a-image-registry-private-configuration\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698496 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/948aab57-f974-48a1-b43a-cb75577e8d7a-registry-certificates\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.698539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698520 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsz8b\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-kube-api-access-fsz8b\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.698929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698564 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-archives-path\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.698929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698587 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-trusted-ca-bundle\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.698929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698623 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-registry-tls\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.698929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698672 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/948aab57-f974-48a1-b43a-cb75577e8d7a-trusted-ca\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.698929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.698696 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/948aab57-f974-48a1-b43a-cb75577e8d7a-installation-pull-secrets\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.699473 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.699446 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9551cad5-7a59-43fe-af7a-ff0ed70a8789-tmp\") pod \"downloads-6dd789c8b7-fmzxl\" (UID: \"9551cad5-7a59-43fe-af7a-ff0ed70a8789\") " pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.700006 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.699981 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-archives-path\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.700153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.700041 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-service-ca-bundle\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.700731 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.700708 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-trusted-ca-bundle\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.702465 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.702444 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/486b788d-7531-41c1-89d5-996d6a05c0d9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7dq5p\" (UID: \"486b788d-7531-41c1-89d5-996d6a05c0d9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" Oct 31 12:32:05.710941 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.710907 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv6fv\" (UniqueName: \"kubernetes.io/projected/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-kube-api-access-lv6fv\") pod \"periodic-gathering-t6v9c-694pl\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.711110 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.711090 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99l8\" (UniqueName: \"kubernetes.io/projected/9551cad5-7a59-43fe-af7a-ff0ed70a8789-kube-api-access-z99l8\") pod \"downloads-6dd789c8b7-fmzxl\" (UID: \"9551cad5-7a59-43fe-af7a-ff0ed70a8789\") " pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.763811 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.763762 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:32:05.787379 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.787346 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" Oct 31 12:32:05.795484 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.795457 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:05.799971 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.799938 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/948aab57-f974-48a1-b43a-cb75577e8d7a-ca-trust-extracted\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800054 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.799990 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-bound-sa-token\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800054 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.800031 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/948aab57-f974-48a1-b43a-cb75577e8d7a-image-registry-private-configuration\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800155 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.800064 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/948aab57-f974-48a1-b43a-cb75577e8d7a-registry-certificates\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800410 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.800383 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsz8b\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-kube-api-access-fsz8b\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800490 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.800474 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-registry-tls\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800546 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.800514 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/948aab57-f974-48a1-b43a-cb75577e8d7a-trusted-ca\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800599 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.800541 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/948aab57-f974-48a1-b43a-cb75577e8d7a-installation-pull-secrets\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.800741 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.800387 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/948aab57-f974-48a1-b43a-cb75577e8d7a-ca-trust-extracted\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.801182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.801157 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/948aab57-f974-48a1-b43a-cb75577e8d7a-registry-certificates\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.801842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.801821 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/948aab57-f974-48a1-b43a-cb75577e8d7a-trusted-ca\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.803403 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.803353 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/948aab57-f974-48a1-b43a-cb75577e8d7a-image-registry-private-configuration\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.803525 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.803510 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/948aab57-f974-48a1-b43a-cb75577e8d7a-installation-pull-secrets\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.804443 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.804400 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-registry-tls\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.813016 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.812955 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-bound-sa-token\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.813212 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.813190 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsz8b\" (UniqueName: \"kubernetes.io/projected/948aab57-f974-48a1-b43a-cb75577e8d7a-kube-api-access-fsz8b\") pod \"image-registry-5cb45b6b6f-8pjrj\" (UID: \"948aab57-f974-48a1-b43a-cb75577e8d7a\") " pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:05.874714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:05.874681 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:06.042072 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.042042 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t6v9c-694pl"] Oct 31 12:32:06.045182 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:32:06.045152 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e27f88c_9de4_4ad9_93d0_bca2279b39ab.slice/crio-5f4e001e71f34c0ba5e14a90b6f157146bb5123471bb1620d2cb0ea61606b62f WatchSource:0}: Error finding container 5f4e001e71f34c0ba5e14a90b6f157146bb5123471bb1620d2cb0ea61606b62f: Status 404 returned error can't find the container with id 5f4e001e71f34c0ba5e14a90b6f157146bb5123471bb1620d2cb0ea61606b62f Oct 31 12:32:06.071145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.071123 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p"] Oct 31 12:32:06.080894 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:32:06.080861 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486b788d_7531_41c1_89d5_996d6a05c0d9.slice/crio-dd835902d57ab6e53d4b0f9f06f0c6dda12753b467c991ca71d808a482828158 WatchSource:0}: Error finding container dd835902d57ab6e53d4b0f9f06f0c6dda12753b467c991ca71d808a482828158: Status 404 returned error can't find the container with id dd835902d57ab6e53d4b0f9f06f0c6dda12753b467c991ca71d808a482828158 Oct 31 12:32:06.100694 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.100669 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fmzxl"] Oct 31 12:32:06.103683 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:32:06.103655 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9551cad5_7a59_43fe_af7a_ff0ed70a8789.slice/crio-58ef67fccfcab60f37b52601ab90820f1ccf264c7822e78df4c2c11d66ca1b9d WatchSource:0}: Error finding container 58ef67fccfcab60f37b52601ab90820f1ccf264c7822e78df4c2c11d66ca1b9d: Status 404 returned error can't find the container with id 58ef67fccfcab60f37b52601ab90820f1ccf264c7822e78df4c2c11d66ca1b9d Oct 31 12:32:06.163501 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.163469 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb45b6b6f-8pjrj"] Oct 31 12:32:06.167757 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:32:06.167731 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948aab57_f974_48a1_b43a_cb75577e8d7a.slice/crio-6a7ff8aa0f077677ccba7db422afe6dd7ecb4e834dc25e10a32b7894e7d79fff WatchSource:0}: Error finding container 6a7ff8aa0f077677ccba7db422afe6dd7ecb4e834dc25e10a32b7894e7d79fff: Status 404 returned error can't find the container with id 6a7ff8aa0f077677ccba7db422afe6dd7ecb4e834dc25e10a32b7894e7d79fff Oct 31 12:32:06.174816 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.174782 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6v9c-694pl" event={"ID":"8e27f88c-9de4-4ad9-93d0-bca2279b39ab","Type":"ContainerStarted","Data":"a3d1dd09d6f0618fe818987eca591fe918908ebddcd631e405c3c8513f93f8d5"} Oct 31 12:32:06.174924 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.174840 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6v9c-694pl" event={"ID":"8e27f88c-9de4-4ad9-93d0-bca2279b39ab","Type":"ContainerStarted","Data":"5f4e001e71f34c0ba5e14a90b6f157146bb5123471bb1620d2cb0ea61606b62f"} Oct 31 12:32:06.176167 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.176140 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fmzxl" event={"ID":"9551cad5-7a59-43fe-af7a-ff0ed70a8789","Type":"ContainerStarted","Data":"58ef67fccfcab60f37b52601ab90820f1ccf264c7822e78df4c2c11d66ca1b9d"} Oct 31 12:32:06.177482 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.177459 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" event={"ID":"948aab57-f974-48a1-b43a-cb75577e8d7a","Type":"ContainerStarted","Data":"6a7ff8aa0f077677ccba7db422afe6dd7ecb4e834dc25e10a32b7894e7d79fff"} Oct 31 12:32:06.178846 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.178821 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" event={"ID":"486b788d-7531-41c1-89d5-996d6a05c0d9","Type":"ContainerStarted","Data":"dd835902d57ab6e53d4b0f9f06f0c6dda12753b467c991ca71d808a482828158"} Oct 31 12:32:06.197622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:06.197579 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t6v9c-694pl" podStartSLOduration=1.197565352 podStartE2EDuration="1.197565352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:06.196416739 +0000 UTC m=+20.374134724" watchObservedRunningTime="2026-10-31 12:32:06.197565352 +0000 UTC m=+20.375283335" Oct 31 12:32:07.185136 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:07.185096 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" event={"ID":"948aab57-f974-48a1-b43a-cb75577e8d7a","Type":"ContainerStarted","Data":"8c95c7c2ec73354d8e0a2bc1b39a0bf014a9cebd0447cc1fb369822a6799e6f6"} Oct 31 12:32:07.185600 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:07.185285 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:07.215149 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:07.214231 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" podStartSLOduration=2.214212286 podStartE2EDuration="2.214212286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:07.212042221 +0000 UTC m=+21.389760238" watchObservedRunningTime="2026-10-31 12:32:07.214212286 +0000 UTC m=+21.391930271" Oct 31 12:32:08.192973 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:08.192932 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" event={"ID":"486b788d-7531-41c1-89d5-996d6a05c0d9","Type":"ContainerStarted","Data":"4c3d0ab5c75b051bf7cb1547234f54b28045fc2098ed22ae706f7394a223613d"} Oct 31 12:32:08.211552 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:08.211501 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" podStartSLOduration=1.799859323 podStartE2EDuration="3.211486868s" totalImagesPullingTime="1.411627545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:08.21033681 +0000 UTC m=+22.388054794" watchObservedRunningTime="2026-10-31 12:32:08.211486868 +0000 UTC m=+22.389204871" Oct 31 12:32:09.198834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:09.198622 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" Oct 31 12:32:09.203982 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:09.203956 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7dq5p" Oct 31 12:32:10.060588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:10.060556 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4psl_8c751f98-9133-4738-9052-8dba6020cda6/dns/0.log" Oct 31 12:32:10.227099 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:10.227065 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4psl_8c751f98-9133-4738-9052-8dba6020cda6/kube-rbac-proxy/0.log" Oct 31 12:32:10.470046 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:10.470016 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:32:10.831320 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:10.831243 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9v4bh_a069e898-8264-4573-b885-2528dadb0305/dns-node-resolver/0.log" Oct 31 12:32:11.026496 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:11.026468 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cb45b6b6f-8pjrj_948aab57-f974-48a1-b43a-cb75577e8d7a/registry/0.log" Oct 31 12:32:11.227980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:11.227956 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f8479bcff-g6cvb_a4942177-472e-47cf-934b-92b58447dfae/registry/0.log" Oct 31 12:32:11.426375 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:11.426344 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85dd98c8f4-j6vtq_300d5937-06ea-46c4-894e-17ab3aea7815/registry/0.log" Oct 31 12:32:11.826789 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:11.826755 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qkx64_0bcc043c-d8ac-4f2a-ae8d-3e233db2842d/node-ca/0.log" Oct 31 12:32:12.226747 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:12.226719 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-865bf4d8bc-d5gwq_579b4046-76c9-4d7e-b77a-9888510a22a8/router/0.log" Oct 31 12:32:12.426622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:12.426581 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqcvc_e040a356-8196-40c6-8f2c-a94bb4c6bfe4/serve-healthcheck-canary/0.log" Oct 31 12:32:14.429139 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.427270 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bgf2c"] Oct 31 12:32:14.456376 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.456348 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.487067 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.486983 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 12:32:14.487257 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.487202 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 12:32:14.487412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.487391 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zhwtd\"" Oct 31 12:32:14.538205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.538175 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 12:32:14.538547 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.538221 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 12:32:14.585708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585664 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.585708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585709 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-sys\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.585882 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585829 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.585882 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585867 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-textfile\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.585957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585898 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-root\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.585957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585919 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-wtmp\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.586031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585960 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xk8\" (UniqueName: \"kubernetes.io/projected/07cde39d-85c9-4b82-9248-c57a806fcbe2-kube-api-access-j5xk8\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.586031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.585987 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07cde39d-85c9-4b82-9248-c57a806fcbe2-metrics-client-ca\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.586031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.586009 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-tls\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.686987 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.686909 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-tls\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.686987 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.686973 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687014 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-sys\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687060 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687090 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-textfile\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687124 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-root\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687134 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-sys\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687144 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-wtmp\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687500 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687225 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xk8\" (UniqueName: \"kubernetes.io/projected/07cde39d-85c9-4b82-9248-c57a806fcbe2-kube-api-access-j5xk8\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687500 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687270 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07cde39d-85c9-4b82-9248-c57a806fcbe2-metrics-client-ca\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687500 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687273 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-wtmp\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687522 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07cde39d-85c9-4b82-9248-c57a806fcbe2-root\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687736 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687683 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.687883 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687853 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-textfile\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.688007 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.687857 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07cde39d-85c9-4b82-9248-c57a806fcbe2-metrics-client-ca\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.689690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.689670 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-tls\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.689987 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.689964 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07cde39d-85c9-4b82-9248-c57a806fcbe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.699520 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.699500 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xk8\" (UniqueName: \"kubernetes.io/projected/07cde39d-85c9-4b82-9248-c57a806fcbe2-kube-api-access-j5xk8\") pod \"node-exporter-bgf2c\" (UID: \"07cde39d-85c9-4b82-9248-c57a806fcbe2\") " pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:14.823530 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:14.823496 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgf2c" Oct 31 12:32:15.226321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:15.226277 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgf2c" event={"ID":"07cde39d-85c9-4b82-9248-c57a806fcbe2","Type":"ContainerStarted","Data":"77205d2c9752673f909e6747796d2ebfcfad5499b9cbb6e39ee30d1af80ae780"} Oct 31 12:32:15.471093 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:15.471062 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:32:16.360153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:16.360123 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8m9k" Oct 31 12:32:17.238579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:17.238538 5074 generic.go:356] "Generic (PLEG): container finished" podID="07cde39d-85c9-4b82-9248-c57a806fcbe2" containerID="b544860d188b1c2c07a34a8bb761bff4df974a02e2b1f32f8da36469cb973c85" exitCode=0 Oct 31 12:32:17.239056 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:17.238612 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgf2c" event={"ID":"07cde39d-85c9-4b82-9248-c57a806fcbe2","Type":"ContainerDied","Data":"b544860d188b1c2c07a34a8bb761bff4df974a02e2b1f32f8da36469cb973c85"} Oct 31 12:32:18.249618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:18.249576 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgf2c" event={"ID":"07cde39d-85c9-4b82-9248-c57a806fcbe2","Type":"ContainerStarted","Data":"467027e586ce109f50f2cd7dc5e49f3fbffd86662207bd4435cea71c027be838"} Oct 31 12:32:18.250168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:18.249666 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgf2c" event={"ID":"07cde39d-85c9-4b82-9248-c57a806fcbe2","Type":"ContainerStarted","Data":"07e3fe9c66aad938ce029dbcfa6992436e1c1b93892d030064a4b55028a6e179"} Oct 31 12:32:18.277217 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:18.277156 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bgf2c" podStartSLOduration=2.6288599980000003 podStartE2EDuration="4.277140394s" totalImagesPullingTime="1.548109009s" totalInitContainerRuntime="100.171387ms" isStatefulPod=true podCreationTimestamp="2026-10-31 12:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:18.274940391 +0000 UTC m=+32.452658376" watchObservedRunningTime="2026-10-31 12:32:18.277140394 +0000 UTC m=+32.454858377" Oct 31 12:32:19.014031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:19.014000 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gjvxs" Oct 31 12:32:25.982016 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:32:25.981972 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:32:26.290243 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:26.290199 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fmzxl" event={"ID":"9551cad5-7a59-43fe-af7a-ff0ed70a8789","Type":"ContainerStarted","Data":"49d753b66a1d963ab60222c45b7e65d7140c593bdf83edc6bdb5e7b11a370e31"} Oct 31 12:32:26.290848 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:26.290809 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:26.303170 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:26.303145 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-fmzxl" Oct 31 12:32:26.328221 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:26.328166 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-fmzxl" podStartSLOduration=1.5882590479999998 podStartE2EDuration="21.328153569s" totalImagesPullingTime="19.739894521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:26.326110905 +0000 UTC m=+40.503828888" watchObservedRunningTime="2026-10-31 12:32:26.328153569 +0000 UTC m=+40.505871555" Oct 31 12:32:28.203939 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:28.203902 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cb45b6b6f-8pjrj" Oct 31 12:32:28.280460 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:28.263006 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8479bcff-g6cvb"] Oct 31 12:32:30.488683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.488605 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" podUID="300d5937-06ea-46c4-894e-17ab3aea7815" containerName="registry" containerID="cri-o://611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b" gracePeriod=30 Oct 31 12:32:30.807184 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.807145 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:32:30.839441 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839406 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.839620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839486 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.839620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839506 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.839620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839535 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.839620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839568 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.839620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839595 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.839920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839626 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.839920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.839742 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\") pod \"300d5937-06ea-46c4-894e-17ab3aea7815\" (UID: \"300d5937-06ea-46c4-894e-17ab3aea7815\") " Oct 31 12:32:30.840737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.840522 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:30.840737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.840663 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:30.849210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.849174 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:30.850240 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.850202 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:30.850575 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.850535 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:30.850819 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.850776 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "kube-api-access-ntcw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:30.851705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.851667 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:30.853153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.853109 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token" pod "300d5937-06ea-46c4-894e-17ab3aea7815" (UID: "300d5937-06ea-46c4-894e-17ab3aea7815"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:30.941304 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941270 5074 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-registry-certificates\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:30.941304 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941301 5074 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/300d5937-06ea-46c4-894e-17ab3aea7815-ca-trust-extracted\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:30.941515 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941316 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntcw7\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-kube-api-access-ntcw7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:30.941515 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941332 5074 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-bound-sa-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:30.941515 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941344 5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/300d5937-06ea-46c4-894e-17ab3aea7815-trusted-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:30.941515 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941353 5074 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-image-registry-private-configuration\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:30.941515 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941363 5074 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/300d5937-06ea-46c4-894e-17ab3aea7815-registry-tls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:30.941515 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:30.941371 5074 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/300d5937-06ea-46c4-894e-17ab3aea7815-installation-pull-secrets\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:31.328955 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.328917 5074 generic.go:356] "Generic (PLEG): container finished" podID="300d5937-06ea-46c4-894e-17ab3aea7815" containerID="611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b" exitCode=0 Oct 31 12:32:31.329157 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.328968 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" Oct 31 12:32:31.329157 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.328991 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" event={"ID":"300d5937-06ea-46c4-894e-17ab3aea7815","Type":"ContainerDied","Data":"611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b"} Oct 31 12:32:31.329157 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.329041 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dd98c8f4-j6vtq" event={"ID":"300d5937-06ea-46c4-894e-17ab3aea7815","Type":"ContainerDied","Data":"5f905b7c80a505b7a0cc1332df363d8f2efb440a75bd40036104d71dd514a7c2"} Oct 31 12:32:31.329157 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.329064 5074 scope.go:122] "RemoveContainer" containerID="611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b" Oct 31 12:32:31.339526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.339506 5074 scope.go:122] "RemoveContainer" containerID="611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b" Oct 31 12:32:31.339930 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:32:31.339895 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b\": container with ID starting with 611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b not found: ID does not exist" containerID="611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b" Oct 31 12:32:31.340051 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.339939 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b"} err="failed to get container status \"611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b\": rpc error: code = NotFound desc = could not find container \"611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b\": container with ID starting with 611f06476725ae6b96e973276f1413f85c3e6314e9887f78154a4403ce449b7b not found: ID does not exist" Oct 31 12:32:31.354450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.354425 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85dd98c8f4-j6vtq"] Oct 31 12:32:31.358994 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.358972 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85dd98c8f4-j6vtq"] Oct 31 12:32:31.944026 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:31.943991 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300d5937-06ea-46c4-894e-17ab3aea7815" path="/var/lib/kubelet/pods/300d5937-06ea-46c4-894e-17ab3aea7815/volumes" Oct 31 12:32:37.936038 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:37.936007 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7vhj5-f2fsm"] Oct 31 12:32:37.936581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:37.936202 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" podUID="55d3094e-4769-4135-9878-d0e72089389e" containerName="insights-gathering" containerID="cri-o://e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1" gracePeriod=30 Oct 31 12:32:38.243021 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.242996 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:32:38.309398 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.309371 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\" (UniqueName: \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\") pod \"55d3094e-4769-4135-9878-d0e72089389e\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " Oct 31 12:32:38.309572 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.309437 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\") pod \"55d3094e-4769-4135-9878-d0e72089389e\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " Oct 31 12:32:38.309572 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.309473 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\") pod \"55d3094e-4769-4135-9878-d0e72089389e\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " Oct 31 12:32:38.309572 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.309510 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\") pod \"55d3094e-4769-4135-9878-d0e72089389e\" (UID: \"55d3094e-4769-4135-9878-d0e72089389e\") " Oct 31 12:32:38.309842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.309814 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle" pod "55d3094e-4769-4135-9878-d0e72089389e" (UID: "55d3094e-4769-4135-9878-d0e72089389e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:38.309842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.309829 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path" pod "55d3094e-4769-4135-9878-d0e72089389e" (UID: "55d3094e-4769-4135-9878-d0e72089389e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:38.309926 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.309901 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle" pod "55d3094e-4769-4135-9878-d0e72089389e" (UID: "55d3094e-4769-4135-9878-d0e72089389e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:38.311773 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.311746 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n" pod "55d3094e-4769-4135-9878-d0e72089389e" (UID: "55d3094e-4769-4135-9878-d0e72089389e"). InnerVolumeSpecName "kube-api-access-h4h8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:38.362285 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.362254 5074 generic.go:356] "Generic (PLEG): container finished" podID="55d3094e-4769-4135-9878-d0e72089389e" containerID="e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1" exitCode=2 Oct 31 12:32:38.362414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.362309 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" Oct 31 12:32:38.362414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.362345 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" event={"ID":"55d3094e-4769-4135-9878-d0e72089389e","Type":"ContainerDied","Data":"e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1"} Oct 31 12:32:38.362414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.362397 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7vhj5-f2fsm" event={"ID":"55d3094e-4769-4135-9878-d0e72089389e","Type":"ContainerDied","Data":"9290f0b0c5766eb9f127a1a5df5c86a8ddca247ef6386ec2b266a8dfc1d0ca29"} Oct 31 12:32:38.362576 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.362417 5074 scope.go:122] "RemoveContainer" containerID="e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1" Oct 31 12:32:38.370895 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.370877 5074 scope.go:122] "RemoveContainer" containerID="e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1" Oct 31 12:32:38.371159 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:32:38.371140 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1\": container with ID starting with e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1 not found: ID does not exist" containerID="e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1" Oct 31 12:32:38.371224 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.371167 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1"} err="failed to get container status \"e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1\": rpc error: code = NotFound desc = could not find container \"e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1\": container with ID starting with e857ab221bac53acb1ea99d0f00aa1349516c690495f05c475a95dbf0c9a2dc1 not found: ID does not exist" Oct 31 12:32:38.382815 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.382793 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7vhj5-f2fsm"] Oct 31 12:32:38.386900 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.386881 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-7vhj5-f2fsm"] Oct 31 12:32:38.410253 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.410222 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4h8n\" (UniqueName: \"kubernetes.io/projected/55d3094e-4769-4135-9878-d0e72089389e-kube-api-access-h4h8n\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:38.410253 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.410252 5074 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-service-ca-bundle\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:38.410395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.410268 5074 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55d3094e-4769-4135-9878-d0e72089389e-archives-path\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:38.410395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:38.410281 5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d3094e-4769-4135-9878-d0e72089389e-trusted-ca-bundle\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:39.935875 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:39.935840 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d3094e-4769-4135-9878-d0e72089389e" path="/var/lib/kubelet/pods/55d3094e-4769-4135-9878-d0e72089389e/volumes" Oct 31 12:32:51.424504 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:51.424463 5074 generic.go:356] "Generic (PLEG): container finished" podID="236577a8-0720-452b-83fa-986cb9aeba14" containerID="7942d4111a4ce9f2a98b6a9f1500821bc4bec5822e47bf43d1c509420eacbc28" exitCode=0 Oct 31 12:32:51.424959 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:51.424541 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" event={"ID":"236577a8-0720-452b-83fa-986cb9aeba14","Type":"ContainerDied","Data":"7942d4111a4ce9f2a98b6a9f1500821bc4bec5822e47bf43d1c509420eacbc28"} Oct 31 12:32:51.425102 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:51.425079 5074 scope.go:122] "RemoveContainer" containerID="7942d4111a4ce9f2a98b6a9f1500821bc4bec5822e47bf43d1c509420eacbc28" Oct 31 12:32:52.429864 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:52.429828 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nggm9" event={"ID":"236577a8-0720-452b-83fa-986cb9aeba14","Type":"ContainerStarted","Data":"1b7fac1c3d3d9a92d4dc1b732373e3346242c045c514a2377377867e1cc8867c"} Oct 31 12:32:53.304829 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.304763 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" podUID="a4942177-472e-47cf-934b-92b58447dfae" containerName="registry" containerID="cri-o://2a303f85cb3b918bd74447013462bfa15c89eec8acab327d0b5cf98b6e8c49d6" gracePeriod=30 Oct 31 12:32:53.435237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.435207 5074 generic.go:356] "Generic (PLEG): container finished" podID="a4942177-472e-47cf-934b-92b58447dfae" containerID="2a303f85cb3b918bd74447013462bfa15c89eec8acab327d0b5cf98b6e8c49d6" exitCode=0 Oct 31 12:32:53.435545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.435289 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" event={"ID":"a4942177-472e-47cf-934b-92b58447dfae","Type":"ContainerDied","Data":"2a303f85cb3b918bd74447013462bfa15c89eec8acab327d0b5cf98b6e8c49d6"} Oct 31 12:32:53.585501 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.585437 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:32:53.742525 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742463 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.742717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742548 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.742717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742583 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.742717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742602 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.742717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742692 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.742943 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742738 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.742943 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742767 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.742943 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.742810 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\") pod \"a4942177-472e-47cf-934b-92b58447dfae\" (UID: \"a4942177-472e-47cf-934b-92b58447dfae\") " Oct 31 12:32:53.743280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.743177 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:53.743649 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.743594 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:32:53.745168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.745139 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:53.745319 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.745297 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:53.745319 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.745308 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:53.745459 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.745418 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "kube-api-access-klpft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:32:53.745496 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.745451 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:32:53.750190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.750160 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted" pod "a4942177-472e-47cf-934b-92b58447dfae" (UID: "a4942177-472e-47cf-934b-92b58447dfae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:32:53.843916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843845 5074 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-installation-pull-secrets\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:53.843916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843875 5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-trusted-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:53.843916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843886 5074 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4942177-472e-47cf-934b-92b58447dfae-ca-trust-extracted\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:53.843916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843896 5074 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4942177-472e-47cf-934b-92b58447dfae-image-registry-private-configuration\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:53.843916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843906 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-kube-api-access-klpft\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:53.843916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843916 5074 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-registry-tls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:53.844189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843924 5074 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4942177-472e-47cf-934b-92b58447dfae-bound-sa-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:53.844189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:53.843933 5074 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4942177-472e-47cf-934b-92b58447dfae-registry-certificates\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:32:54.440963 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:54.440923 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" event={"ID":"a4942177-472e-47cf-934b-92b58447dfae","Type":"ContainerDied","Data":"dc1b0ef4305e4b51daf6d10e35406e186e94d6522f2da274fd7268f84d8a121b"} Oct 31 12:32:54.441426 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:54.440982 5074 scope.go:122] "RemoveContainer" containerID="2a303f85cb3b918bd74447013462bfa15c89eec8acab327d0b5cf98b6e8c49d6" Oct 31 12:32:54.441426 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:54.440940 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8479bcff-g6cvb" Oct 31 12:32:54.460166 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:54.460142 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8479bcff-g6cvb"] Oct 31 12:32:54.464031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:54.464009 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f8479bcff-g6cvb"] Oct 31 12:32:55.936737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:32:55.936709 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4942177-472e-47cf-934b-92b58447dfae" path="/var/lib/kubelet/pods/a4942177-472e-47cf-934b-92b58447dfae/volumes" Oct 31 12:32:55.981462 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:32:55.981428 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:33:25.980995 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:33:25.980958 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:33:31.594134 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:31.594038 5074 generic.go:356] "Generic (PLEG): container finished" podID="8e27f88c-9de4-4ad9-93d0-bca2279b39ab" containerID="a3d1dd09d6f0618fe818987eca591fe918908ebddcd631e405c3c8513f93f8d5" exitCode=0 Oct 31 12:33:31.594662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:31.594116 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6v9c-694pl" event={"ID":"8e27f88c-9de4-4ad9-93d0-bca2279b39ab","Type":"ContainerDied","Data":"a3d1dd09d6f0618fe818987eca591fe918908ebddcd631e405c3c8513f93f8d5"} Oct 31 12:33:32.727188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.727161 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:33:32.745193 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.745163 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-trusted-ca-bundle\") pod \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " Oct 31 12:33:32.745711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.745227 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-service-ca-bundle\") pod \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " Oct 31 12:33:32.745711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.745587 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-service-ca-bundle" pod "8e27f88c-9de4-4ad9-93d0-bca2279b39ab" (UID: "8e27f88c-9de4-4ad9-93d0-bca2279b39ab"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:33:32.745711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.745692 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-kube-api-access-lv6fv\" (UniqueName: \"kubernetes.io/projected/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-kube-api-access-lv6fv\") pod \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " Oct 31 12:33:32.745857 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.745733 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-archives-path\") pod \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\" (UID: \"8e27f88c-9de4-4ad9-93d0-bca2279b39ab\") " Oct 31 12:33:32.746139 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.746100 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-archives-path" pod "8e27f88c-9de4-4ad9-93d0-bca2279b39ab" (UID: "8e27f88c-9de4-4ad9-93d0-bca2279b39ab"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:33:32.746386 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.746351 5074 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-archives-path\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:32.746386 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.746382 5074 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-service-ca-bundle\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:32.746625 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.746491 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-trusted-ca-bundle" pod "8e27f88c-9de4-4ad9-93d0-bca2279b39ab" (UID: "8e27f88c-9de4-4ad9-93d0-bca2279b39ab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:33:32.748034 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.748006 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-kube-api-access-lv6fv" pod "8e27f88c-9de4-4ad9-93d0-bca2279b39ab" (UID: "8e27f88c-9de4-4ad9-93d0-bca2279b39ab"). InnerVolumeSpecName "kube-api-access-lv6fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:33:32.847225 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.847187 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv6fv\" (UniqueName: \"kubernetes.io/projected/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-kube-api-access-lv6fv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:32.847225 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:32.847220 5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e27f88c-9de4-4ad9-93d0-bca2279b39ab-trusted-ca-bundle\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:33:33.603854 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:33.603816 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t6v9c-694pl" Oct 31 12:33:33.603854 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:33.603825 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t6v9c-694pl" event={"ID":"8e27f88c-9de4-4ad9-93d0-bca2279b39ab","Type":"ContainerDied","Data":"5f4e001e71f34c0ba5e14a90b6f157146bb5123471bb1620d2cb0ea61606b62f"} Oct 31 12:33:33.604122 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:33:33.603866 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f4e001e71f34c0ba5e14a90b6f157146bb5123471bb1620d2cb0ea61606b62f" Oct 31 12:33:55.980758 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:33:55.980719 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:34:25.981778 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:34:25.981739 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:34:55.980794 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:34:55.980751 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:35:25.981179 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:35:25.981145 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:35:55.980120 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:35:55.980083 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:36:15.246262 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246225 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb"] Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246542 5074 cpu_manager.go:414] "containerMap: removing container" podUID="55d3094e-4769-4135-9878-d0e72089389e" containerName="insights-gathering" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246552 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d3094e-4769-4135-9878-d0e72089389e" containerName="insights-gathering" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246562 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a4942177-472e-47cf-934b-92b58447dfae" containerName="registry" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246567 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4942177-472e-47cf-934b-92b58447dfae" containerName="registry" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246594 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8e27f88c-9de4-4ad9-93d0-bca2279b39ab" containerName="insights-gathering" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246601 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e27f88c-9de4-4ad9-93d0-bca2279b39ab" containerName="insights-gathering" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246609 5074 cpu_manager.go:414] "containerMap: removing container" podUID="300d5937-06ea-46c4-894e-17ab3aea7815" containerName="registry" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246614 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300d5937-06ea-46c4-894e-17ab3aea7815" containerName="registry" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246680 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e27f88c-9de4-4ad9-93d0-bca2279b39ab" containerName="insights-gathering" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246688 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="300d5937-06ea-46c4-894e-17ab3aea7815" containerName="registry" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246696 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d3094e-4769-4135-9878-d0e72089389e" containerName="insights-gathering" Oct 31 12:36:15.247723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.246701 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4942177-472e-47cf-934b-92b58447dfae" containerName="registry" Oct 31 12:36:15.248734 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.248717 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.255443 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.255419 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 12:36:15.256716 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.256698 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 12:36:15.256807 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.256699 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-nt2s4\"" Oct 31 12:36:15.261467 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.261444 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb"] Oct 31 12:36:15.315557 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.315518 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.315753 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.315589 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.315753 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.315703 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/6f7032a9-5270-40cd-9b8d-957763c6c5fc-kube-api-access-lsl88\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.416099 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.416057 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.416271 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.416110 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/6f7032a9-5270-40cd-9b8d-957763c6c5fc-kube-api-access-lsl88\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.416271 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.416155 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.416462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.416438 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.416526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.416504 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.432206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.432171 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/6f7032a9-5270-40cd-9b8d-957763c6c5fc-kube-api-access-lsl88\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.561160 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.561073 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:15.765727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.765696 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh"] Oct 31 12:36:15.768036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.768014 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.783668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.783622 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 12:36:15.784904 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.784870 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2bzjw\"" Oct 31 12:36:15.785759 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.785736 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh"] Oct 31 12:36:15.821342 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821275 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821342 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821309 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821342 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821337 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821582 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821446 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821582 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821495 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp8sc\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-kube-api-access-zp8sc\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821582 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821554 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821582 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821574 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821734 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821600 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.821734 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.821621 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.868246 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.868217 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb"] Oct 31 12:36:15.870921 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:36:15.870895 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7032a9_5270_40cd_9b8d_957763c6c5fc.slice/crio-fec78c6498cfee9cd54284c6a394a55078653c4dc7e6331784dcd5b396bfca29 WatchSource:0}: Error finding container fec78c6498cfee9cd54284c6a394a55078653c4dc7e6331784dcd5b396bfca29: Status 404 returned error can't find the container with id fec78c6498cfee9cd54284c6a394a55078653c4dc7e6331784dcd5b396bfca29 Oct 31 12:36:15.922331 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922295 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922331 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922334 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922373 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922397 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922426 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp8sc\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-kube-api-access-zp8sc\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922475 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922502 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922546 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922912 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922586 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.922912 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922874 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.923018 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922924 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.923018 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.922953 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.923130 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.923081 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.923505 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.923478 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.925482 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.925463 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.925834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.925776 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.941312 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.941284 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk"] Oct 31 12:36:15.944422 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.944400 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:15.948704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.948592 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp8sc\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-kube-api-access-zp8sc\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.950041 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.950021 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-28rh7\"" Oct 31 12:36:15.950538 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.950516 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-97tmh\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:15.964029 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:15.964001 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk"] Oct 31 12:36:16.023928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.023892 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20266055-9cae-476d-8268-8f20300ff729-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.023928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.023934 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.024122 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.023955 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.024162 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.024117 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.024199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.024170 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.024284 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.024270 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.024319 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.024293 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20266055-9cae-476d-8268-8f20300ff729-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.024319 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.024309 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20266055-9cae-476d-8268-8f20300ff729-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.024425 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.024338 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrsk\" (UniqueName: \"kubernetes.io/projected/20266055-9cae-476d-8268-8f20300ff729-kube-api-access-rxrsk\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.080728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.080645 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:16.125730 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.125695 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.125888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.125739 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.125888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.125843 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.125888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.125881 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126011 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.125960 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126056 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126008 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20266055-9cae-476d-8268-8f20300ff729-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126056 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126034 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20266055-9cae-476d-8268-8f20300ff729-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126064 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrsk\" (UniqueName: \"kubernetes.io/projected/20266055-9cae-476d-8268-8f20300ff729-kube-api-access-rxrsk\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126106 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20266055-9cae-476d-8268-8f20300ff729-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126266 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126229 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126325 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126306 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126344 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126657 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20266055-9cae-476d-8268-8f20300ff729-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.126909 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.126887 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.128763 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.128733 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20266055-9cae-476d-8268-8f20300ff729-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.128899 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.128758 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20266055-9cae-476d-8268-8f20300ff729-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.140954 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.140925 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20266055-9cae-476d-8268-8f20300ff729-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.141947 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.141923 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrsk\" (UniqueName: \"kubernetes.io/projected/20266055-9cae-476d-8268-8f20300ff729-kube-api-access-rxrsk\") pod \"openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk\" (UID: \"20266055-9cae-476d-8268-8f20300ff729\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.232292 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.232255 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" event={"ID":"6f7032a9-5270-40cd-9b8d-957763c6c5fc","Type":"ContainerStarted","Data":"fec78c6498cfee9cd54284c6a394a55078653c4dc7e6331784dcd5b396bfca29"} Oct 31 12:36:16.258507 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.258456 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:16.365891 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.365835 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh"] Oct 31 12:36:16.368538 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:36:16.368510 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd37002_4c0c_4b16_957b_06bc5ed9c447.slice/crio-3a817df39ff8c2c33851e27e8611e7eac0a04c62b39ee178a59406251d58f0b3 WatchSource:0}: Error finding container 3a817df39ff8c2c33851e27e8611e7eac0a04c62b39ee178a59406251d58f0b3: Status 404 returned error can't find the container with id 3a817df39ff8c2c33851e27e8611e7eac0a04c62b39ee178a59406251d58f0b3 Oct 31 12:36:16.539732 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:16.539680 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk"] Oct 31 12:36:16.545485 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:36:16.545455 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20266055_9cae_476d_8268_8f20300ff729.slice/crio-44a101de03cc33783896ad1946fb681b554c2410960186c1933c874d30f4c05c WatchSource:0}: Error finding container 44a101de03cc33783896ad1946fb681b554c2410960186c1933c874d30f4c05c: Status 404 returned error can't find the container with id 44a101de03cc33783896ad1946fb681b554c2410960186c1933c874d30f4c05c Oct 31 12:36:17.237955 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:17.237905 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" event={"ID":"20266055-9cae-476d-8268-8f20300ff729","Type":"ContainerStarted","Data":"44a101de03cc33783896ad1946fb681b554c2410960186c1933c874d30f4c05c"} Oct 31 12:36:17.239360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:17.239331 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" event={"ID":"0bd37002-4c0c-4b16-957b-06bc5ed9c447","Type":"ContainerStarted","Data":"3a817df39ff8c2c33851e27e8611e7eac0a04c62b39ee178a59406251d58f0b3"} Oct 31 12:36:20.313123 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:20.313085 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:36:20.313536 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:20.313178 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:36:20.313536 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:20.313227 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:36:20.358769 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:20.358719 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:36:20.358874 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:20.358819 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:36:20.358874 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:20.358863 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:36:21.263452 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:21.263408 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" event={"ID":"0bd37002-4c0c-4b16-957b-06bc5ed9c447","Type":"ContainerStarted","Data":"88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59"} Oct 31 12:36:21.265194 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:21.265161 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" event={"ID":"20266055-9cae-476d-8268-8f20300ff729","Type":"ContainerStarted","Data":"5531fb831d371c29b33a0df0c357adf8526068086571a0aa5d0fdde2ff3d9126"} Oct 31 12:36:21.292062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:21.292004 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podStartSLOduration=2.304715559 podStartE2EDuration="6.291989115s" totalImagesPullingTime="3.987273556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:21.291541494 +0000 UTC m=+275.469259480" watchObservedRunningTime="2026-10-31 12:36:21.291989115 +0000 UTC m=+275.469707099" Oct 31 12:36:21.315191 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:21.315131 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" podStartSLOduration=2.550826985 podStartE2EDuration="6.31511576s" totalImagesPullingTime="3.764288775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:21.314168776 +0000 UTC m=+275.491886772" watchObservedRunningTime="2026-10-31 12:36:21.31511576 +0000 UTC m=+275.492833744" Oct 31 12:36:22.081103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.081066 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:22.086783 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.086751 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:22.259453 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.259415 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:22.264620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.264589 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:22.271273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.271239 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" event={"ID":"6f7032a9-5270-40cd-9b8d-957763c6c5fc","Type":"ContainerStarted","Data":"66f0cf96ddd3a654cbb2a882ce86348da2b0febc8026bde06e002890dca18cfe"} Oct 31 12:36:22.272148 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.271568 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:22.272276 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.272183 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:22.272497 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.272470 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk" Oct 31 12:36:22.272598 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:22.272535 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:36:23.277290 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:23.277255 5074 generic.go:356] "Generic (PLEG): container finished" podID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerID="66f0cf96ddd3a654cbb2a882ce86348da2b0febc8026bde06e002890dca18cfe" exitCode=0 Oct 31 12:36:23.277742 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:23.277294 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" event={"ID":"6f7032a9-5270-40cd-9b8d-957763c6c5fc","Type":"ContainerDied","Data":"66f0cf96ddd3a654cbb2a882ce86348da2b0febc8026bde06e002890dca18cfe"} Oct 31 12:36:24.283362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:24.283326 5074 generic.go:356] "Generic (PLEG): container finished" podID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerID="6ed7363eb97acab8f8f39a52eadf44c4e927617c6dc571268c6e8fdbf477bfa7" exitCode=0 Oct 31 12:36:24.283766 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:24.283402 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" event={"ID":"6f7032a9-5270-40cd-9b8d-957763c6c5fc","Type":"ContainerDied","Data":"6ed7363eb97acab8f8f39a52eadf44c4e927617c6dc571268c6e8fdbf477bfa7"} Oct 31 12:36:25.985691 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:36:25.985653 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:36:34.329231 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:34.329188 5074 generic.go:356] "Generic (PLEG): container finished" podID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerID="157394567f0c2383af6cc258656f7faf7960391adc43b6757507355a45cc96c6" exitCode=0 Oct 31 12:36:34.329679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:34.329288 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" event={"ID":"6f7032a9-5270-40cd-9b8d-957763c6c5fc","Type":"ContainerDied","Data":"157394567f0c2383af6cc258656f7faf7960391adc43b6757507355a45cc96c6"} Oct 31 12:36:35.459307 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.459281 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:35.598433 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.598355 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-bundle\") pod \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " Oct 31 12:36:35.598433 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.598416 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7032a9-5270-40cd-9b8d-957763c6c5fc-kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/6f7032a9-5270-40cd-9b8d-957763c6c5fc-kube-api-access-lsl88\") pod \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " Oct 31 12:36:35.598610 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.598455 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-util\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-util\") pod \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\" (UID: \"6f7032a9-5270-40cd-9b8d-957763c6c5fc\") " Oct 31 12:36:35.599008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.598978 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-bundle" pod "6f7032a9-5270-40cd-9b8d-957763c6c5fc" (UID: "6f7032a9-5270-40cd-9b8d-957763c6c5fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:35.600729 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.600701 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7032a9-5270-40cd-9b8d-957763c6c5fc-kube-api-access-lsl88" pod "6f7032a9-5270-40cd-9b8d-957763c6c5fc" (UID: "6f7032a9-5270-40cd-9b8d-957763c6c5fc"). InnerVolumeSpecName "kube-api-access-lsl88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:36:35.602655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.602619 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-util" pod "6f7032a9-5270-40cd-9b8d-957763c6c5fc" (UID: "6f7032a9-5270-40cd-9b8d-957763c6c5fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:36:35.699235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.699198 5074 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-util\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:35.699235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.699229 5074 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f7032a9-5270-40cd-9b8d-957763c6c5fc-bundle\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:35.699235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:35.699239 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsl88\" (UniqueName: \"kubernetes.io/projected/6f7032a9-5270-40cd-9b8d-957763c6c5fc-kube-api-access-lsl88\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:36:36.340568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:36.340538 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" Oct 31 12:36:36.340568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:36.340547 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ttkzb" event={"ID":"6f7032a9-5270-40cd-9b8d-957763c6c5fc","Type":"ContainerDied","Data":"fec78c6498cfee9cd54284c6a394a55078653c4dc7e6331784dcd5b396bfca29"} Oct 31 12:36:36.340794 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:36.340592 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fec78c6498cfee9cd54284c6a394a55078653c4dc7e6331784dcd5b396bfca29" Oct 31 12:36:43.301274 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.301237 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk"] Oct 31 12:36:43.302519 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.302491 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerName="pull" Oct 31 12:36:43.302681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.302667 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerName="pull" Oct 31 12:36:43.302796 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.302786 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerName="util" Oct 31 12:36:43.302872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.302863 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerName="util" Oct 31 12:36:43.302947 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.302938 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerName="extract" Oct 31 12:36:43.303013 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.303003 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerName="extract" Oct 31 12:36:43.303282 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.303268 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7032a9-5270-40cd-9b8d-957763c6c5fc" containerName="extract" Oct 31 12:36:43.308385 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.308364 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.333039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.333019 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 12:36:43.333523 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.333505 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 12:36:43.336447 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:36:43.336420 5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" is forbidden: User \"system:node:ip-10-0-135-41.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub-operator-system\": no relationship found between node 'ip-10-0-135-41.us-east-2.compute.internal' and this object" podUID="3716314e-4607-45e8-898d-fe7505c8f7fe" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.374542 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:36:43.374508 5074 reflector.go:204] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-135-41.us-east-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub-operator-system\": no relationship found between node 'ip-10-0-135-41.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Oct 31 12:36:43.374542 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:36:43.374524 5074 reflector.go:204] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-96llj\" is forbidden: User \"system:node:ip-10-0-135-41.us-east-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub-operator-system\": no relationship found between node 'ip-10-0-135-41.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-96llj\"" type="*v1.Secret" Oct 31 12:36:43.377647 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.377618 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 12:36:43.412404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.412380 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 12:36:43.466179 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.466150 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3716314e-4607-45e8-898d-fe7505c8f7fe-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.466291 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.466192 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.466291 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.466237 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3716314e-4607-45e8-898d-fe7505c8f7fe-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.466389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.466289 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.466389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.466342 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.466389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.466378 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf5g\" (UniqueName: \"kubernetes.io/projected/3716314e-4607-45e8-898d-fe7505c8f7fe-kube-api-access-9jf5g\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.534797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.534765 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk"] Oct 31 12:36:43.566868 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.566799 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.566868 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.566840 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.567066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.566871 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf5g\" (UniqueName: \"kubernetes.io/projected/3716314e-4607-45e8-898d-fe7505c8f7fe-kube-api-access-9jf5g\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.567066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.566908 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3716314e-4607-45e8-898d-fe7505c8f7fe-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.567066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.566932 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.567066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.566962 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3716314e-4607-45e8-898d-fe7505c8f7fe-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.567421 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.567391 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3716314e-4607-45e8-898d-fe7505c8f7fe-manifests\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.569424 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.569391 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3716314e-4607-45e8-898d-fe7505c8f7fe-tmp\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.569513 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.569462 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.569614 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.569597 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.569672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.569603 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3716314e-4607-45e8-898d-fe7505c8f7fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:43.585134 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:43.585112 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf5g\" (UniqueName: \"kubernetes.io/projected/3716314e-4607-45e8-898d-fe7505c8f7fe-kube-api-access-9jf5g\") pod \"opendatahub-operator-controller-manager-5c44b469d4-lm5gk\" (UID: \"3716314e-4607-45e8-898d-fe7505c8f7fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:44.539716 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:44.539680 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 12:36:44.621201 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:44.621165 5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" secret="" err="failed to sync secret cache: timed out waiting for the condition" Oct 31 12:36:44.621358 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:44.621250 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:44.638543 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:44.638520 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-96llj\"" Oct 31 12:36:44.937528 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:36:44.937502 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3716314e_4607_45e8_898d_fe7505c8f7fe.slice/crio-1ce015755d56da0c5acc9112399253af04a542a3832125e1bb1009bf65fb1749 WatchSource:0}: Error finding container 1ce015755d56da0c5acc9112399253af04a542a3832125e1bb1009bf65fb1749: Status 404 returned error can't find the container with id 1ce015755d56da0c5acc9112399253af04a542a3832125e1bb1009bf65fb1749 Oct 31 12:36:44.939743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:44.939722 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk"] Oct 31 12:36:45.377108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:45.377074 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" event={"ID":"3716314e-4607-45e8-898d-fe7505c8f7fe","Type":"ContainerStarted","Data":"1ce015755d56da0c5acc9112399253af04a542a3832125e1bb1009bf65fb1749"} Oct 31 12:36:45.964427 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:45.964399 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:36:45.965088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:45.965063 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:36:48.392650 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:48.392605 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" event={"ID":"3716314e-4607-45e8-898d-fe7505c8f7fe","Type":"ContainerStarted","Data":"9f0731c66a4c4f7adbc34fc3fa25985c5315b651e8eee2ede4957b5fc762e817"} Oct 31 12:36:49.398623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:49.398590 5074 generic.go:356] "Generic (PLEG): container finished" podID="3716314e-4607-45e8-898d-fe7505c8f7fe" containerID="9f0731c66a4c4f7adbc34fc3fa25985c5315b651e8eee2ede4957b5fc762e817" exitCode=0 Oct 31 12:36:49.399093 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:49.398671 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" event={"ID":"3716314e-4607-45e8-898d-fe7505c8f7fe","Type":"ContainerDied","Data":"9f0731c66a4c4f7adbc34fc3fa25985c5315b651e8eee2ede4957b5fc762e817"} Oct 31 12:36:49.399275 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:49.399248 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:36:50.411135 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:50.411097 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" event={"ID":"3716314e-4607-45e8-898d-fe7505c8f7fe","Type":"ContainerStarted","Data":"5fb7439d88cd8f5551a00398ff2572ad9734f7a971db254b2bda259d2827baa6"} Oct 31 12:36:50.411555 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:50.411233 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:36:50.472152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:36:50.472099 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" podStartSLOduration=3.389126268 podStartE2EDuration="7.472084179s" totalImagesPullingTime="3.63556122s" totalInitContainerRuntime="447.396691ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:50.469332065 +0000 UTC m=+304.647050049" watchObservedRunningTime="2026-10-31 12:36:50.472084179 +0000 UTC m=+304.649802163" Oct 31 12:36:55.981103 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:36:55.981064 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:37:01.417628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:01.417597 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c44b469d4-lm5gk" Oct 31 12:37:09.076878 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.076847 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx"] Oct 31 12:37:09.097269 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.097239 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx"] Oct 31 12:37:09.097419 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.097326 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.105591 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.105570 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 12:37:09.105758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.105705 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 12:37:09.105758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.105736 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-58c4z\"" Oct 31 12:37:09.190606 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.190577 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c041f2d-517a-464f-827d-e7778a4c4775-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6smdx\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.190767 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.190618 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxr8l\" (UniqueName: \"kubernetes.io/projected/5c041f2d-517a-464f-827d-e7778a4c4775-kube-api-access-cxr8l\") pod \"kuadrant-operator-controller-manager-678d87766f-6smdx\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.291348 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.291314 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c041f2d-517a-464f-827d-e7778a4c4775-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6smdx\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.291534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.291364 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxr8l\" (UniqueName: \"kubernetes.io/projected/5c041f2d-517a-464f-827d-e7778a4c4775-kube-api-access-cxr8l\") pod \"kuadrant-operator-controller-manager-678d87766f-6smdx\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.291800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.291776 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c041f2d-517a-464f-827d-e7778a4c4775-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6smdx\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.331334 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.331276 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxr8l\" (UniqueName: \"kubernetes.io/projected/5c041f2d-517a-464f-827d-e7778a4c4775-kube-api-access-cxr8l\") pod \"kuadrant-operator-controller-manager-678d87766f-6smdx\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.414691 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.414657 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:09.704155 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:09.704129 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx"] Oct 31 12:37:09.706606 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:37:09.706576 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c041f2d_517a_464f_827d_e7778a4c4775.slice/crio-2230bbd5137fb052d1fe146f7d755d99bff98af24a0a3d56e1458e7a069a51e5 WatchSource:0}: Error finding container 2230bbd5137fb052d1fe146f7d755d99bff98af24a0a3d56e1458e7a069a51e5: Status 404 returned error can't find the container with id 2230bbd5137fb052d1fe146f7d755d99bff98af24a0a3d56e1458e7a069a51e5 Oct 31 12:37:10.497124 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:10.497084 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" event={"ID":"5c041f2d-517a-464f-827d-e7778a4c4775","Type":"ContainerStarted","Data":"2230bbd5137fb052d1fe146f7d755d99bff98af24a0a3d56e1458e7a069a51e5"} Oct 31 12:37:14.324075 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.324044 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb"] Oct 31 12:37:14.326845 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.326824 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" Oct 31 12:37:14.330708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.330686 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x4mpw\"" Oct 31 12:37:14.340085 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.340059 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb"] Oct 31 12:37:14.439304 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.439273 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqtm\" (UniqueName: \"kubernetes.io/projected/6673fd3f-afa0-4106-901a-7afcad6cdeee-kube-api-access-mkqtm\") pod \"authorino-operator-7b7bb4bc77-r6gnb\" (UID: \"6673fd3f-afa0-4106-901a-7afcad6cdeee\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" Oct 31 12:37:14.540674 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.540623 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqtm\" (UniqueName: \"kubernetes.io/projected/6673fd3f-afa0-4106-901a-7afcad6cdeee-kube-api-access-mkqtm\") pod \"authorino-operator-7b7bb4bc77-r6gnb\" (UID: \"6673fd3f-afa0-4106-901a-7afcad6cdeee\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" Oct 31 12:37:14.557854 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.557829 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqtm\" (UniqueName: \"kubernetes.io/projected/6673fd3f-afa0-4106-901a-7afcad6cdeee-kube-api-access-mkqtm\") pod \"authorino-operator-7b7bb4bc77-r6gnb\" (UID: \"6673fd3f-afa0-4106-901a-7afcad6cdeee\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" Oct 31 12:37:14.638347 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.638276 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" Oct 31 12:37:14.909152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:14.909128 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb"] Oct 31 12:37:14.912069 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:37:14.912041 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6673fd3f_afa0_4106_901a_7afcad6cdeee.slice/crio-d69ec41785df957584486ea16ea31d53105457c17c1c32adf262416c22c13122 WatchSource:0}: Error finding container d69ec41785df957584486ea16ea31d53105457c17c1c32adf262416c22c13122: Status 404 returned error can't find the container with id d69ec41785df957584486ea16ea31d53105457c17c1c32adf262416c22c13122 Oct 31 12:37:15.519406 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:15.519370 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" event={"ID":"6673fd3f-afa0-4106-901a-7afcad6cdeee","Type":"ContainerStarted","Data":"d69ec41785df957584486ea16ea31d53105457c17c1c32adf262416c22c13122"} Oct 31 12:37:18.534760 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:18.534720 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" event={"ID":"6673fd3f-afa0-4106-901a-7afcad6cdeee","Type":"ContainerStarted","Data":"ff3c7ebe953e8f7648ced64cb88101e9f2fdfb234fb521a83c446d7e3fead8c2"} Oct 31 12:37:18.535131 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:18.534843 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" Oct 31 12:37:18.556656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:18.556585 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" podStartSLOduration=1.931763278 podStartE2EDuration="4.55657138s" totalImagesPullingTime="2.624808102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:18.554171161 +0000 UTC m=+332.731889147" watchObservedRunningTime="2026-10-31 12:37:18.55657138 +0000 UTC m=+332.734289363" Oct 31 12:37:19.872766 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.872732 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq"] Oct 31 12:37:19.880823 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.880798 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.893869 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.893841 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq"] Oct 31 12:37:19.895469 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.895445 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-99h7t\"" Oct 31 12:37:19.992731 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.992696 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2ss\" (UniqueName: \"kubernetes.io/projected/7284a125-b30e-41ef-b402-9d43ae1e75ad-kube-api-access-jb2ss\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.992731 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.992737 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.992981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.992764 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.992981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.992876 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.992981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.992919 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-data\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.992981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.992959 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.993121 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.993055 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-token\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.993121 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.993113 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284a125-b30e-41ef-b402-9d43ae1e75ad-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:19.993194 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:19.993172 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.094404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094364 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.094404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094406 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094425 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-data\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094445 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094494 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-token\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094540 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284a125-b30e-41ef-b402-9d43ae1e75ad-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094580 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094652 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2ss\" (UniqueName: \"kubernetes.io/projected/7284a125-b30e-41ef-b402-9d43ae1e75ad-kube-api-access-jb2ss\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094687 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.094968 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.095388 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095554 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.095504 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095554 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.095525 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-data\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.095987 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.095959 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284a125-b30e-41ef-b402-9d43ae1e75ad-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.096966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.096900 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.097449 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.097424 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.109095 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.109069 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284a125-b30e-41ef-b402-9d43ae1e75ad-istio-token\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.109518 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.109487 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2ss\" (UniqueName: \"kubernetes.io/projected/7284a125-b30e-41ef-b402-9d43ae1e75ad-kube-api-access-jb2ss\") pod \"data-science-gateway-data-science-gateway-class-657b74449cczhbq\" (UID: \"7284a125-b30e-41ef-b402-9d43ae1e75ad\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.196074 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.195983 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:20.502893 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.502870 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq"] Oct 31 12:37:20.505048 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:37:20.505019 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7284a125_b30e_41ef_b402_9d43ae1e75ad.slice/crio-215816d5b8a7c54957579349678c5758c5f6786fa455458979c74fd1cbf950da WatchSource:0}: Error finding container 215816d5b8a7c54957579349678c5758c5f6786fa455458979c74fd1cbf950da: Status 404 returned error can't find the container with id 215816d5b8a7c54957579349678c5758c5f6786fa455458979c74fd1cbf950da Oct 31 12:37:20.507290 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.507254 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:37:20.507395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.507335 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:37:20.507395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.507380 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:37:20.544462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:20.544433 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" event={"ID":"7284a125-b30e-41ef-b402-9d43ae1e75ad","Type":"ContainerStarted","Data":"215816d5b8a7c54957579349678c5758c5f6786fa455458979c74fd1cbf950da"} Oct 31 12:37:21.551015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:21.550978 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" event={"ID":"7284a125-b30e-41ef-b402-9d43ae1e75ad","Type":"ContainerStarted","Data":"73b6086e8dcd17efd2b84a101e7b733eae6ab4d0d822cf8474d60c71ff905c6b"} Oct 31 12:37:21.585263 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:21.585190 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" podStartSLOduration=2.5851696889999998 podStartE2EDuration="2.585169689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:21.584389747 +0000 UTC m=+335.762107731" watchObservedRunningTime="2026-10-31 12:37:21.585169689 +0000 UTC m=+335.762887675" Oct 31 12:37:22.196788 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:22.196751 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:22.202258 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:22.202232 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:22.556282 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:22.556248 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:22.557354 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:22.557328 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-657b74449cczhbq" Oct 31 12:37:23.030783 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.030734 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx"] Oct 31 12:37:23.033409 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.033387 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.043237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.043158 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-c97h4\"" Oct 31 12:37:23.043237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.043206 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 12:37:23.043530 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.043504 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 12:37:23.046362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.046147 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx"] Oct 31 12:37:23.047484 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.047462 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 12:37:23.126573 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.126533 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39defbd-ec41-4ab3-a577-ba969686c319-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.126756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.126643 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vr6h\" (UniqueName: \"kubernetes.io/projected/c39defbd-ec41-4ab3-a577-ba969686c319-kube-api-access-6vr6h\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.131762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.126792 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c39defbd-ec41-4ab3-a577-ba969686c319-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.131762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.126926 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c39defbd-ec41-4ab3-a577-ba969686c319-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.227629 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.227579 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c39defbd-ec41-4ab3-a577-ba969686c319-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.227817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.227698 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39defbd-ec41-4ab3-a577-ba969686c319-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.227817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.227726 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vr6h\" (UniqueName: \"kubernetes.io/projected/c39defbd-ec41-4ab3-a577-ba969686c319-kube-api-access-6vr6h\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.227817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.227776 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c39defbd-ec41-4ab3-a577-ba969686c319-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.228062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.228033 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c39defbd-ec41-4ab3-a577-ba969686c319-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.228151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.228112 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39defbd-ec41-4ab3-a577-ba969686c319-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.228818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.228795 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c39defbd-ec41-4ab3-a577-ba969686c319-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.240159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.240136 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vr6h\" (UniqueName: \"kubernetes.io/projected/c39defbd-ec41-4ab3-a577-ba969686c319-kube-api-access-6vr6h\") pod \"kserve-module-controller-manager-7566cf5fc8-tpdfx\" (UID: \"c39defbd-ec41-4ab3-a577-ba969686c319\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.349769 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.349694 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:23.629568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:23.629534 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx"] Oct 31 12:37:23.632241 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:37:23.632210 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39defbd_ec41_4ab3_a577_ba969686c319.slice/crio-5ff9989a44c634996c602adc9b2b7f504151b9cd97484ccaee8784c8888e15ae WatchSource:0}: Error finding container 5ff9989a44c634996c602adc9b2b7f504151b9cd97484ccaee8784c8888e15ae: Status 404 returned error can't find the container with id 5ff9989a44c634996c602adc9b2b7f504151b9cd97484ccaee8784c8888e15ae Oct 31 12:37:24.568400 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:24.568315 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" event={"ID":"c39defbd-ec41-4ab3-a577-ba969686c319","Type":"ContainerStarted","Data":"5ff9989a44c634996c602adc9b2b7f504151b9cd97484ccaee8784c8888e15ae"} Oct 31 12:37:24.572175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:24.571784 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" event={"ID":"5c041f2d-517a-464f-827d-e7778a4c4775","Type":"ContainerStarted","Data":"abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571"} Oct 31 12:37:24.572175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:24.571874 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:24.601474 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:24.601412 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" podStartSLOduration=1.010195638 podStartE2EDuration="15.601390278s" totalImagesPullingTime="14.59119464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:24.596664939 +0000 UTC m=+338.774382965" watchObservedRunningTime="2026-10-31 12:37:24.601390278 +0000 UTC m=+338.779108264" Oct 31 12:37:25.982175 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:37:25.982140 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:37:26.588376 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:26.588336 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" event={"ID":"c39defbd-ec41-4ab3-a577-ba969686c319","Type":"ContainerStarted","Data":"0b2c506b8a5284fa4b049783e1c7a4584b37ac4047e144ee5af17404ef6f43a9"} Oct 31 12:37:26.588554 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:26.588444 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:26.611926 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:26.611850 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" podStartSLOduration=1.340132912 podStartE2EDuration="3.611838686s" totalImagesPullingTime="2.271705774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:26.610436254 +0000 UTC m=+340.788154273" watchObservedRunningTime="2026-10-31 12:37:26.611838686 +0000 UTC m=+340.789556691" Oct 31 12:37:29.541076 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:29.541044 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r6gnb" Oct 31 12:37:35.578888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:35.578857 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:37:47.594030 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:37:47.593995 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tpdfx" Oct 31 12:37:55.983819 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:37:55.983774 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:38:17.038321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.038283 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst"] Oct 31 12:38:17.045248 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.045227 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.060094 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.060068 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst"] Oct 31 12:38:17.107093 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.107065 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 12:38:17.107357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.107333 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b7295\"" Oct 31 12:38:17.217023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.216990 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssmc\" (UniqueName: \"kubernetes.io/projected/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-kube-api-access-jssmc\") pod \"llmisvc-controller-manager-869d5dcc5d-8bhst\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.217189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.217126 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8bhst\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.318442 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.318370 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8bhst\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.318442 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.318428 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssmc\" (UniqueName: \"kubernetes.io/projected/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-kube-api-access-jssmc\") pod \"llmisvc-controller-manager-869d5dcc5d-8bhst\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.321061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.321033 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8bhst\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.331072 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.331045 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssmc\" (UniqueName: \"kubernetes.io/projected/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-kube-api-access-jssmc\") pod \"llmisvc-controller-manager-869d5dcc5d-8bhst\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.359066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.359043 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:17.674029 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.674002 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst"] Oct 31 12:38:17.675992 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:38:17.675960 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1715eeea_93f5_4ba6_8b0e_c1ca3c6691d5.slice/crio-fe2d209bf4a4a04a22eaa0cc4392bbcf3c004a4046a160a0a9e8524dc4135462 WatchSource:0}: Error finding container fe2d209bf4a4a04a22eaa0cc4392bbcf3c004a4046a160a0a9e8524dc4135462: Status 404 returned error can't find the container with id fe2d209bf4a4a04a22eaa0cc4392bbcf3c004a4046a160a0a9e8524dc4135462 Oct 31 12:38:17.826084 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:17.826048 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" event={"ID":"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5","Type":"ContainerStarted","Data":"fe2d209bf4a4a04a22eaa0cc4392bbcf3c004a4046a160a0a9e8524dc4135462"} Oct 31 12:38:20.841362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:20.841320 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" event={"ID":"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5","Type":"ContainerStarted","Data":"6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e"} Oct 31 12:38:20.841818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:20.841444 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:20.860519 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:20.860466 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" podStartSLOduration=1.266240027 podStartE2EDuration="3.860452423s" totalImagesPullingTime="2.594212396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:20.86032046 +0000 UTC m=+395.038038447" watchObservedRunningTime="2026-10-31 12:38:20.860452423 +0000 UTC m=+395.038170408" Oct 31 12:38:25.982665 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:38:25.982615 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:38:51.848582 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:38:51.848547 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:38:55.981409 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:38:55.981371 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:39:25.981842 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:39:25.981802 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:39:55.981711 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:39:55.981675 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:40:25.980677 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:40:25.980622 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:40:55.981316 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:40:55.981281 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:41:25.981512 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:41:25.981476 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:41:45.993155 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:41:45.993121 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:41:45.995210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:41:45.995185 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:41:55.981087 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:41:55.981049 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:42:25.982324 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:42:25.982287 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:42:55.981095 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:42:55.981058 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:43:25.980609 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:43:25.980574 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:43:55.981674 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:43:55.981622 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:44:25.981732 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:44:25.981697 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:44:55.981501 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:44:55.981455 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:45:25.982000 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:45:25.981946 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:45:55.981417 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:45:55.981383 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:46:25.981328 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:46:25.981284 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:46:46.021152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:46:46.021122 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:46:46.025427 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:46:46.025402 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:46:55.981543 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:46:55.981507 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:47:25.985396 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:47:25.985351 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:47:39.842168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:39.842136 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6smdx_5c041f2d-517a-464f-827d-e7778a4c4775/manager/0.log" Oct 31 12:47:40.041782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.041748 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx"] Oct 31 12:47:40.041983 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.041941 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" podUID="5c041f2d-517a-464f-827d-e7778a4c4775" containerName="manager" containerID="cri-o://abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571" gracePeriod=10 Oct 31 12:47:40.074796 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.074770 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j"] Oct 31 12:47:40.078429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.078408 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.093847 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.093788 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j"] Oct 31 12:47:40.158968 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.158938 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dfb15b-fa89-461c-b36a-bceb692300c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jsr6j\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.159099 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.159047 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bcw\" (UniqueName: \"kubernetes.io/projected/87dfb15b-fa89-461c-b36a-bceb692300c0-kube-api-access-d2bcw\") pod \"kuadrant-operator-controller-manager-678d87766f-jsr6j\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.259819 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.259779 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dfb15b-fa89-461c-b36a-bceb692300c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jsr6j\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.259976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.259897 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bcw\" (UniqueName: \"kubernetes.io/projected/87dfb15b-fa89-461c-b36a-bceb692300c0-kube-api-access-d2bcw\") pod \"kuadrant-operator-controller-manager-678d87766f-jsr6j\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.260140 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.260119 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dfb15b-fa89-461c-b36a-bceb692300c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jsr6j\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.272817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.272790 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bcw\" (UniqueName: \"kubernetes.io/projected/87dfb15b-fa89-461c-b36a-bceb692300c0-kube-api-access-d2bcw\") pod \"kuadrant-operator-controller-manager-678d87766f-jsr6j\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.296300 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.296279 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:47:40.360846 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.360779 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c041f2d-517a-464f-827d-e7778a4c4775-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c041f2d-517a-464f-827d-e7778a4c4775-extensions-socket-volume\") pod \"5c041f2d-517a-464f-827d-e7778a4c4775\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " Oct 31 12:47:40.361010 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.360868 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c041f2d-517a-464f-827d-e7778a4c4775-kube-api-access-cxr8l\" (UniqueName: \"kubernetes.io/projected/5c041f2d-517a-464f-827d-e7778a4c4775-kube-api-access-cxr8l\") pod \"5c041f2d-517a-464f-827d-e7778a4c4775\" (UID: \"5c041f2d-517a-464f-827d-e7778a4c4775\") " Oct 31 12:47:40.361257 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.361226 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c041f2d-517a-464f-827d-e7778a4c4775-extensions-socket-volume" pod "5c041f2d-517a-464f-827d-e7778a4c4775" (UID: "5c041f2d-517a-464f-827d-e7778a4c4775"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:47:40.363281 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.363254 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c041f2d-517a-464f-827d-e7778a4c4775-kube-api-access-cxr8l" pod "5c041f2d-517a-464f-827d-e7778a4c4775" (UID: "5c041f2d-517a-464f-827d-e7778a4c4775"). InnerVolumeSpecName "kube-api-access-cxr8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:47:40.363539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.363515 5074 generic.go:356] "Generic (PLEG): container finished" podID="5c041f2d-517a-464f-827d-e7778a4c4775" containerID="abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571" exitCode=0 Oct 31 12:47:40.363600 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.363580 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" Oct 31 12:47:40.363685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.363601 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" event={"ID":"5c041f2d-517a-464f-827d-e7778a4c4775","Type":"ContainerDied","Data":"abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571"} Oct 31 12:47:40.363727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.363691 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx" event={"ID":"5c041f2d-517a-464f-827d-e7778a4c4775","Type":"ContainerDied","Data":"2230bbd5137fb052d1fe146f7d755d99bff98af24a0a3d56e1458e7a069a51e5"} Oct 31 12:47:40.363769 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.363724 5074 scope.go:122] "RemoveContainer" containerID="abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571" Oct 31 12:47:40.377994 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.377975 5074 scope.go:122] "RemoveContainer" containerID="abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571" Oct 31 12:47:40.378293 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:47:40.378274 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571\": container with ID starting with abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571 not found: ID does not exist" containerID="abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571" Oct 31 12:47:40.378352 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.378303 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571"} err="failed to get container status \"abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571\": rpc error: code = NotFound desc = could not find container \"abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571\": container with ID starting with abffe92316cf6c30f552dee3fc038fc8e4d54aed0b25dd4da4519c7400794571 not found: ID does not exist" Oct 31 12:47:40.391116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.391096 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx"] Oct 31 12:47:40.398979 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.398960 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6smdx"] Oct 31 12:47:40.425521 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.425499 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:40.462086 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.462051 5074 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c041f2d-517a-464f-827d-e7778a4c4775-extensions-socket-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:40.462086 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.462089 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxr8l\" (UniqueName: \"kubernetes.io/projected/5c041f2d-517a-464f-827d-e7778a4c4775-kube-api-access-cxr8l\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:47:40.707147 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:40.707123 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j"] Oct 31 12:47:40.709354 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:47:40.709323 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dfb15b_fa89_461c_b36a_bceb692300c0.slice/crio-460b355d6592e5e3c2e7d8eee5e818bd08cc03bb4256ad709c5c51f7eb93bcb7 WatchSource:0}: Error finding container 460b355d6592e5e3c2e7d8eee5e818bd08cc03bb4256ad709c5c51f7eb93bcb7: Status 404 returned error can't find the container with id 460b355d6592e5e3c2e7d8eee5e818bd08cc03bb4256ad709c5c51f7eb93bcb7 Oct 31 12:47:41.369510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:41.369471 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" event={"ID":"87dfb15b-fa89-461c-b36a-bceb692300c0","Type":"ContainerStarted","Data":"91369d75f398460a3e9e1b0e44645d6884f03af6bafec1b8d5672a80bb693649"} Oct 31 12:47:41.369977 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:41.369528 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" event={"ID":"87dfb15b-fa89-461c-b36a-bceb692300c0","Type":"ContainerStarted","Data":"460b355d6592e5e3c2e7d8eee5e818bd08cc03bb4256ad709c5c51f7eb93bcb7"} Oct 31 12:47:41.369977 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:41.369568 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:41.394019 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:41.393956 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" podStartSLOduration=1.393940922 podStartE2EDuration="1.393940922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:41.392477736 +0000 UTC m=+955.570195720" watchObservedRunningTime="2026-10-31 12:47:41.393940922 +0000 UTC m=+955.571658907" Oct 31 12:47:41.938012 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:41.937974 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c041f2d-517a-464f-827d-e7778a4c4775" path="/var/lib/kubelet/pods/5c041f2d-517a-464f-827d-e7778a4c4775/volumes" Oct 31 12:47:52.378137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:47:52.378108 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 12:47:55.980912 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:47:55.980878 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:48:00.440607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.440573 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:48:00.441100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.440946 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5c041f2d-517a-464f-827d-e7778a4c4775" containerName="manager" Oct 31 12:48:00.441100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.440960 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c041f2d-517a-464f-827d-e7778a4c4775" containerName="manager" Oct 31 12:48:00.441100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.441048 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c041f2d-517a-464f-827d-e7778a4c4775" containerName="manager" Oct 31 12:48:00.445318 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.445296 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:00.452772 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.452749 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rdq67\"" Oct 31 12:48:00.452888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.452822 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:48:00.482558 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.482511 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 12:48:00.484577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.484552 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:48:00.533047 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.533023 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfszd\" (UniqueName: \"kubernetes.io/projected/fbeeade1-58fc-47fb-b398-6547ad67129b-kube-api-access-nfszd\") pod \"limitador-limitador-7c9b87d574-6jt82\" (UID: \"fbeeade1-58fc-47fb-b398-6547ad67129b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:00.533190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.533071 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbeeade1-58fc-47fb-b398-6547ad67129b-config-file\") pod \"limitador-limitador-7c9b87d574-6jt82\" (UID: \"fbeeade1-58fc-47fb-b398-6547ad67129b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:00.633948 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.633905 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbeeade1-58fc-47fb-b398-6547ad67129b-config-file\") pod \"limitador-limitador-7c9b87d574-6jt82\" (UID: \"fbeeade1-58fc-47fb-b398-6547ad67129b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:00.634117 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.634001 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfszd\" (UniqueName: \"kubernetes.io/projected/fbeeade1-58fc-47fb-b398-6547ad67129b-kube-api-access-nfszd\") pod \"limitador-limitador-7c9b87d574-6jt82\" (UID: \"fbeeade1-58fc-47fb-b398-6547ad67129b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:00.634617 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.634591 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbeeade1-58fc-47fb-b398-6547ad67129b-config-file\") pod \"limitador-limitador-7c9b87d574-6jt82\" (UID: \"fbeeade1-58fc-47fb-b398-6547ad67129b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:00.646059 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.646027 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfszd\" (UniqueName: \"kubernetes.io/projected/fbeeade1-58fc-47fb-b398-6547ad67129b-kube-api-access-nfszd\") pod \"limitador-limitador-7c9b87d574-6jt82\" (UID: \"fbeeade1-58fc-47fb-b398-6547ad67129b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:00.759852 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:00.759821 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:01.032475 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:01.032447 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:48:01.035156 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:48:01.035128 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeeade1_58fc_47fb_b398_6547ad67129b.slice/crio-6947841225c4234c6617baca51f07809956f64f64934911b3f5564248a66af1c WatchSource:0}: Error finding container 6947841225c4234c6617baca51f07809956f64f64934911b3f5564248a66af1c: Status 404 returned error can't find the container with id 6947841225c4234c6617baca51f07809956f64f64934911b3f5564248a66af1c Oct 31 12:48:01.037137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:01.037092 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:48:01.463434 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:01.463350 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" event={"ID":"fbeeade1-58fc-47fb-b398-6547ad67129b","Type":"ContainerStarted","Data":"6947841225c4234c6617baca51f07809956f64f64934911b3f5564248a66af1c"} Oct 31 12:48:02.469684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:02.469651 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" event={"ID":"fbeeade1-58fc-47fb-b398-6547ad67129b","Type":"ContainerStarted","Data":"49e832ecf8aa83e58442c61ab8ec263c35f4c13d98c32c72edcaa6b1d403c981"} Oct 31 12:48:02.470167 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:02.469764 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:02.488917 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:02.488870 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" podStartSLOduration=1.200035231 podStartE2EDuration="2.488856504s" totalImagesPullingTime="1.288821273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:02.488406988 +0000 UTC m=+976.666124973" watchObservedRunningTime="2026-10-31 12:48:02.488856504 +0000 UTC m=+976.666574487" Oct 31 12:48:13.475128 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:13.475098 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6jt82" Oct 31 12:48:17.936132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:17.936101 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vsfcv-rjmh5"] Oct 31 12:48:17.941770 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:17.941743 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:17.943262 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:17.943234 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vsfcv-rjmh5"] Oct 31 12:48:17.979109 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:17.979079 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-service-ca-bundle\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:17.979240 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:17.979118 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-trusted-ca-bundle\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:17.979287 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:17.979268 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wsc\" (UniqueName: \"kubernetes.io/projected/1e503e6f-5c7e-432f-b495-090af8eaf6dd-kube-api-access-25wsc\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:17.979323 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:17.979297 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1e503e6f-5c7e-432f-b495-090af8eaf6dd-archives-path\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.080770 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.080735 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-service-ca-bundle\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.080953 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.080782 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-trusted-ca-bundle\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.080953 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.080823 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wsc\" (UniqueName: \"kubernetes.io/projected/1e503e6f-5c7e-432f-b495-090af8eaf6dd-kube-api-access-25wsc\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.080953 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.080843 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1e503e6f-5c7e-432f-b495-090af8eaf6dd-archives-path\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.081246 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.081226 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1e503e6f-5c7e-432f-b495-090af8eaf6dd-archives-path\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.081523 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.081499 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-service-ca-bundle\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.081593 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.081521 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-trusted-ca-bundle\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.093460 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.093443 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wsc\" (UniqueName: \"kubernetes.io/projected/1e503e6f-5c7e-432f-b495-090af8eaf6dd-kube-api-access-25wsc\") pod \"periodic-gathering-vsfcv-rjmh5\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.254847 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.254824 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:48:18.529185 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.529054 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vsfcv-rjmh5"] Oct 31 12:48:18.532003 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:48:18.531975 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e503e6f_5c7e_432f_b495_090af8eaf6dd.slice/crio-4bbab5e31d4f24c2fc91f09729b0b5af117ff1fdbe185cc5be27850f61761b6e WatchSource:0}: Error finding container 4bbab5e31d4f24c2fc91f09729b0b5af117ff1fdbe185cc5be27850f61761b6e: Status 404 returned error can't find the container with id 4bbab5e31d4f24c2fc91f09729b0b5af117ff1fdbe185cc5be27850f61761b6e Oct 31 12:48:18.542358 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:18.542325 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" event={"ID":"1e503e6f-5c7e-432f-b495-090af8eaf6dd","Type":"ContainerStarted","Data":"4bbab5e31d4f24c2fc91f09729b0b5af117ff1fdbe185cc5be27850f61761b6e"} Oct 31 12:48:19.548948 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:19.548914 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" event={"ID":"1e503e6f-5c7e-432f-b495-090af8eaf6dd","Type":"ContainerStarted","Data":"24ac1386b69ab93f87f73c8a54ea58b9786495b292e8b56f94baec1d4bb8e594"} Oct 31 12:48:19.569622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:48:19.569569 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" podStartSLOduration=2.569556712 podStartE2EDuration="2.569556712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:19.567834414 +0000 UTC m=+993.745552398" watchObservedRunningTime="2026-10-31 12:48:19.569556712 +0000 UTC m=+993.747274695" Oct 31 12:48:25.982100 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:48:25.982060 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:48:55.981472 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:48:55.981434 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:49:20.380615 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:20.380580 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:49:26.003843 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:49:26.003802 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:49:30.909605 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:30.909564 5074 generic.go:356] "Generic (PLEG): container finished" podID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerID="88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59" exitCode=0 Oct 31 12:49:30.910070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:30.909610 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" event={"ID":"0bd37002-4c0c-4b16-957b-06bc5ed9c447","Type":"ContainerDied","Data":"88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59"} Oct 31 12:49:30.910070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:30.910003 5074 scope.go:122] "RemoveContainer" containerID="88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59" Oct 31 12:49:30.911239 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:30.911205 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:49:30.911297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:30.911279 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:49:30.911332 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:30.911312 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:49:31.916732 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:31.916701 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" event={"ID":"0bd37002-4c0c-4b16-957b-06bc5ed9c447","Type":"ContainerStarted","Data":"01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5"} Oct 31 12:49:32.095699 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:32.094060 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:49:32.095699 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:32.094195 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-97tmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 12:49:32.095699 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:32.094241 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 12:49:33.090658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:33.090095 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-97tmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 12:49:33.090658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:33.090179 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 12:49:34.089659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:34.089060 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-97tmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 12:49:34.089659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:34.089124 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 12:49:35.083177 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.083131 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-97tmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 12:49:35.083836 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.083795 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 12:49:35.300217 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.300169 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb"] Oct 31 12:49:35.303601 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.303582 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.312869 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.312842 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb"] Oct 31 12:49:35.431417 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.431319 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrw6v\" (UniqueName: \"kubernetes.io/projected/a830a12a-e88b-4cca-97f6-8aa69759dfbd-kube-api-access-zrw6v\") pod \"llmisvc-controller-manager-6466c554d8-xtpgb\" (UID: \"a830a12a-e88b-4cca-97f6-8aa69759dfbd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.431746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.431721 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a830a12a-e88b-4cca-97f6-8aa69759dfbd-cert\") pod \"llmisvc-controller-manager-6466c554d8-xtpgb\" (UID: \"a830a12a-e88b-4cca-97f6-8aa69759dfbd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.533039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.532995 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrw6v\" (UniqueName: \"kubernetes.io/projected/a830a12a-e88b-4cca-97f6-8aa69759dfbd-kube-api-access-zrw6v\") pod \"llmisvc-controller-manager-6466c554d8-xtpgb\" (UID: \"a830a12a-e88b-4cca-97f6-8aa69759dfbd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.533210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.533068 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a830a12a-e88b-4cca-97f6-8aa69759dfbd-cert\") pod \"llmisvc-controller-manager-6466c554d8-xtpgb\" (UID: \"a830a12a-e88b-4cca-97f6-8aa69759dfbd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.535562 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.535529 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a830a12a-e88b-4cca-97f6-8aa69759dfbd-cert\") pod \"llmisvc-controller-manager-6466c554d8-xtpgb\" (UID: \"a830a12a-e88b-4cca-97f6-8aa69759dfbd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.546882 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.546854 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrw6v\" (UniqueName: \"kubernetes.io/projected/a830a12a-e88b-4cca-97f6-8aa69759dfbd-kube-api-access-zrw6v\") pod \"llmisvc-controller-manager-6466c554d8-xtpgb\" (UID: \"a830a12a-e88b-4cca-97f6-8aa69759dfbd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.616593 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.616551 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:35.918923 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.918691 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb"] Oct 31 12:49:35.921281 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:49:35.921249 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda830a12a_e88b_4cca_97f6_8aa69759dfbd.slice/crio-a685ebdb69d3c26d23ae4b899f47c00bf2d3d14edc4e4e5f998bd8df97ee1fa6 WatchSource:0}: Error finding container a685ebdb69d3c26d23ae4b899f47c00bf2d3d14edc4e4e5f998bd8df97ee1fa6: Status 404 returned error can't find the container with id a685ebdb69d3c26d23ae4b899f47c00bf2d3d14edc4e4e5f998bd8df97ee1fa6 Oct 31 12:49:35.937765 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:35.937734 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" event={"ID":"a830a12a-e88b-4cca-97f6-8aa69759dfbd","Type":"ContainerStarted","Data":"a685ebdb69d3c26d23ae4b899f47c00bf2d3d14edc4e4e5f998bd8df97ee1fa6"} Oct 31 12:49:36.090657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:36.090094 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-97tmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 12:49:36.090657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:36.090161 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 12:49:36.944197 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:36.944159 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" event={"ID":"a830a12a-e88b-4cca-97f6-8aa69759dfbd","Type":"ContainerStarted","Data":"33106c26f699685c3f03379eeb0eeb6f5de7027e10e10a910cbf6190569a297c"} Oct 31 12:49:36.944390 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:36.944293 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:49:36.965264 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:36.965203 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" podStartSLOduration=1.585967208 podStartE2EDuration="1.965189974s" totalImagesPullingTime="379.222766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:36.964789635 +0000 UTC m=+1071.142507620" watchObservedRunningTime="2026-10-31 12:49:36.965189974 +0000 UTC m=+1071.142907958" Oct 31 12:49:37.085073 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:37.085042 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:49:37.085445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:37.085425 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:49:37.086050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:37.086030 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 12:49:52.012857 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:52.012823 5074 generic.go:356] "Generic (PLEG): container finished" podID="1e503e6f-5c7e-432f-b495-090af8eaf6dd" containerID="24ac1386b69ab93f87f73c8a54ea58b9786495b292e8b56f94baec1d4bb8e594" exitCode=0 Oct 31 12:49:52.013277 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:52.012896 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" event={"ID":"1e503e6f-5c7e-432f-b495-090af8eaf6dd","Type":"ContainerDied","Data":"24ac1386b69ab93f87f73c8a54ea58b9786495b292e8b56f94baec1d4bb8e594"} Oct 31 12:49:53.157458 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.157429 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:49:53.184244 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.184216 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e503e6f-5c7e-432f-b495-090af8eaf6dd-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1e503e6f-5c7e-432f-b495-090af8eaf6dd-archives-path\") pod \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " Oct 31 12:49:53.184390 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.184293 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-service-ca-bundle\") pod \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " Oct 31 12:49:53.184390 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.184346 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e503e6f-5c7e-432f-b495-090af8eaf6dd-kube-api-access-25wsc\" (UniqueName: \"kubernetes.io/projected/1e503e6f-5c7e-432f-b495-090af8eaf6dd-kube-api-access-25wsc\") pod \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " Oct 31 12:49:53.184461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.184388 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-trusted-ca-bundle\") pod \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\" (UID: \"1e503e6f-5c7e-432f-b495-090af8eaf6dd\") " Oct 31 12:49:53.184587 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.184560 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e503e6f-5c7e-432f-b495-090af8eaf6dd-archives-path" pod "1e503e6f-5c7e-432f-b495-090af8eaf6dd" (UID: "1e503e6f-5c7e-432f-b495-090af8eaf6dd"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:49:53.184743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.184716 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-service-ca-bundle" pod "1e503e6f-5c7e-432f-b495-090af8eaf6dd" (UID: "1e503e6f-5c7e-432f-b495-090af8eaf6dd"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:49:53.184859 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.184836 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-trusted-ca-bundle" pod "1e503e6f-5c7e-432f-b495-090af8eaf6dd" (UID: "1e503e6f-5c7e-432f-b495-090af8eaf6dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:49:53.186498 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.186478 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e503e6f-5c7e-432f-b495-090af8eaf6dd-kube-api-access-25wsc" pod "1e503e6f-5c7e-432f-b495-090af8eaf6dd" (UID: "1e503e6f-5c7e-432f-b495-090af8eaf6dd"). InnerVolumeSpecName "kube-api-access-25wsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:49:53.285563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.285477 5074 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1e503e6f-5c7e-432f-b495-090af8eaf6dd-archives-path\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:53.285563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.285510 5074 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-service-ca-bundle\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:53.285563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.285520 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wsc\" (UniqueName: \"kubernetes.io/projected/1e503e6f-5c7e-432f-b495-090af8eaf6dd-kube-api-access-25wsc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:53.285563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:53.285530 5074 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e503e6f-5c7e-432f-b495-090af8eaf6dd-trusted-ca-bundle\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:49:54.023272 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:54.023239 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" Oct 31 12:49:54.023444 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:54.023271 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vsfcv-rjmh5" event={"ID":"1e503e6f-5c7e-432f-b495-090af8eaf6dd","Type":"ContainerDied","Data":"4bbab5e31d4f24c2fc91f09729b0b5af117ff1fdbe185cc5be27850f61761b6e"} Oct 31 12:49:54.023444 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:49:54.023319 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bbab5e31d4f24c2fc91f09729b0b5af117ff1fdbe185cc5be27850f61761b6e" Oct 31 12:49:55.982483 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:49:55.982448 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:50:07.951558 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:07.951528 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xtpgb" Oct 31 12:50:08.005744 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.005709 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst"] Oct 31 12:50:08.005973 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.005947 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" podUID="1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" containerName="manager" containerID="cri-o://6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e" gracePeriod=30 Oct 31 12:50:08.256656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.256617 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:50:08.316459 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.316424 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-kube-api-access-jssmc\" (UniqueName: \"kubernetes.io/projected/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-kube-api-access-jssmc\") pod \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " Oct 31 12:50:08.316661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.316500 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-cert\" (UniqueName: \"kubernetes.io/secret/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-cert\") pod \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\" (UID: \"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5\") " Oct 31 12:50:08.318757 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.318713 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-cert" pod "1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" (UID: "1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:50:08.318857 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.318781 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-kube-api-access-jssmc" pod "1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" (UID: "1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5"). InnerVolumeSpecName "kube-api-access-jssmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:50:08.417587 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.417526 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jssmc\" (UniqueName: \"kubernetes.io/projected/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-kube-api-access-jssmc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:50:08.417587 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:08.417579 5074 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:50:09.094627 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.094589 5074 generic.go:356] "Generic (PLEG): container finished" podID="1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" containerID="6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e" exitCode=0 Oct 31 12:50:09.095135 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.094652 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" Oct 31 12:50:09.095135 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.094677 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" event={"ID":"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5","Type":"ContainerDied","Data":"6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e"} Oct 31 12:50:09.095135 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.094731 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst" event={"ID":"1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5","Type":"ContainerDied","Data":"fe2d209bf4a4a04a22eaa0cc4392bbcf3c004a4046a160a0a9e8524dc4135462"} Oct 31 12:50:09.095135 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.094755 5074 scope.go:122] "RemoveContainer" containerID="6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e" Oct 31 12:50:09.104589 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.104566 5074 scope.go:122] "RemoveContainer" containerID="6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e" Oct 31 12:50:09.104901 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:50:09.104882 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e\": container with ID starting with 6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e not found: ID does not exist" containerID="6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e" Oct 31 12:50:09.104966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.104909 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e"} err="failed to get container status \"6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e\": rpc error: code = NotFound desc = could not find container \"6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e\": container with ID starting with 6207c00a71a2f8248367983bb22d8c5b727eaf8d9526ff29e8fc31cf0772fd6e not found: ID does not exist" Oct 31 12:50:09.119026 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.118999 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst"] Oct 31 12:50:09.125705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.125677 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8bhst"] Oct 31 12:50:09.936551 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:09.936511 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" path="/var/lib/kubelet/pods/1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5/volumes" Oct 31 12:50:25.981530 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:50:25.981486 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:50:44.061787 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:44.061751 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:50:48.385343 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:48.385306 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:50:51.366728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:51.366686 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:50:54.951552 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:54.951518 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:50:55.995402 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:50:55.995352 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:50:59.476372 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:50:59.476319 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:51:16.906317 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.906276 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg"] Oct 31 12:51:16.906864 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.906799 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1e503e6f-5c7e-432f-b495-090af8eaf6dd" containerName="insights-gathering" Oct 31 12:51:16.906864 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.906816 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e503e6f-5c7e-432f-b495-090af8eaf6dd" containerName="insights-gathering" Oct 31 12:51:16.906864 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.906835 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" containerName="manager" Oct 31 12:51:16.906864 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.906842 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" containerName="manager" Oct 31 12:51:16.907012 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.906923 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e503e6f-5c7e-432f-b495-090af8eaf6dd" containerName="insights-gathering" Oct 31 12:51:16.907012 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.906930 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1715eeea-93f5-4ba6-8b0e-c1ca3c6691d5" containerName="manager" Oct 31 12:51:16.909980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.909959 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:16.923434 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:16.923407 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg"] Oct 31 12:51:17.055881 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.055849 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19865d77-5b0f-456a-a09e-b1922459c50a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056068 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.055893 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.056062 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.056131 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056247 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.056167 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056298 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.056242 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19865d77-5b0f-456a-a09e-b1922459c50a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056298 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.056280 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tph7\" (UniqueName: \"kubernetes.io/projected/19865d77-5b0f-456a-a09e-b1922459c50a-kube-api-access-5tph7\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056447 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.056420 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19865d77-5b0f-456a-a09e-b1922459c50a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.056499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.056484 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157118 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157030 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157118 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157066 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157118 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157082 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157118 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157110 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19865d77-5b0f-456a-a09e-b1922459c50a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157150 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tph7\" (UniqueName: \"kubernetes.io/projected/19865d77-5b0f-456a-a09e-b1922459c50a-kube-api-access-5tph7\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157207 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19865d77-5b0f-456a-a09e-b1922459c50a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157247 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157310 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19865d77-5b0f-456a-a09e-b1922459c50a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157344 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157484 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157551 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157754 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157722 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.157754 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.157746 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.158044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.158025 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19865d77-5b0f-456a-a09e-b1922459c50a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.159457 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.159424 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19865d77-5b0f-456a-a09e-b1922459c50a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.159797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.159774 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19865d77-5b0f-456a-a09e-b1922459c50a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.169821 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.169795 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19865d77-5b0f-456a-a09e-b1922459c50a-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.169946 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.169921 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tph7\" (UniqueName: \"kubernetes.io/projected/19865d77-5b0f-456a-a09e-b1922459c50a-kube-api-access-5tph7\") pod \"maas-default-gateway-openshift-default-fc8c56547-5wltg\" (UID: \"19865d77-5b0f-456a-a09e-b1922459c50a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.223685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.223649 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:17.506540 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:17.506509 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19865d77_5b0f_456a_a09e_b1922459c50a.slice/crio-1510867adbffee7289a2e71de0997d469b06602ea045d7562a5d4eac3eeb2575 WatchSource:0}: Error finding container 1510867adbffee7289a2e71de0997d469b06602ea045d7562a5d4eac3eeb2575: Status 404 returned error can't find the container with id 1510867adbffee7289a2e71de0997d469b06602ea045d7562a5d4eac3eeb2575 Oct 31 12:51:17.509156 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.509118 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:17.509308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.509203 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:17.509308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.509250 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:17.511740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:17.511714 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg"] Oct 31 12:51:18.456084 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:18.456049 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" event={"ID":"19865d77-5b0f-456a-a09e-b1922459c50a","Type":"ContainerStarted","Data":"d6c5f72064518ea8693c0c5e2f753744145184c9c9a94f03510a7644bfe0629e"} Oct 31 12:51:18.456502 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:18.456100 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" event={"ID":"19865d77-5b0f-456a-a09e-b1922459c50a","Type":"ContainerStarted","Data":"1510867adbffee7289a2e71de0997d469b06602ea045d7562a5d4eac3eeb2575"} Oct 31 12:51:18.489080 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:18.489004 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podStartSLOduration=2.488983031 podStartE2EDuration="2.488983031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:18.486159496 +0000 UTC m=+1172.663877482" watchObservedRunningTime="2026-10-31 12:51:18.488983031 +0000 UTC m=+1172.666701017" Oct 31 12:51:19.224287 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:19.224244 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:19.225884 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:19.225855 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 12:51:19.226015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:19.225908 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 12:51:20.224549 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:20.224505 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 12:51:20.224991 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:20.224569 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 12:51:21.224033 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:21.223994 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 12:51:21.224195 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:21.224084 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 12:51:22.224247 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:22.224209 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 12:51:22.224709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:22.224297 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 12:51:23.225021 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:23.224981 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 12:51:23.225499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:23.225041 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 12:51:24.224497 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:24.224463 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 12:51:24.224695 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:24.224532 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 12:51:25.715470 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:25.715432 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:25.716008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:25.715971 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:25.716826 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:25.716780 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" Oct 31 12:51:25.982813 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:51:25.982729 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:51:46.064165 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.064134 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:51:46.067164 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.067141 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:51:46.198985 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.198949 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v"] Oct 31 12:51:46.202626 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.202599 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.210545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.210519 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cf7e6604\"" Oct 31 12:51:46.211851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.211826 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cf7e6604-dockercfg-p74c5\"" Oct 31 12:51:46.213613 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.213583 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v"] Oct 31 12:51:46.354930 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.354848 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e00f41-6042-44e5-b33e-f9373f5d1c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.355308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.355275 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gjz\" (UniqueName: \"kubernetes.io/projected/39e00f41-6042-44e5-b33e-f9373f5d1c95-kube-api-access-76gjz\") pod \"payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.456036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.456000 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gjz\" (UniqueName: \"kubernetes.io/projected/39e00f41-6042-44e5-b33e-f9373f5d1c95-kube-api-access-76gjz\") pod \"payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.456220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.456062 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e00f41-6042-44e5-b33e-f9373f5d1c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.456883 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.456862 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e00f41-6042-44e5-b33e-f9373f5d1c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.515889 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.515857 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gjz\" (UniqueName: \"kubernetes.io/projected/39e00f41-6042-44e5-b33e-f9373f5d1c95-kube-api-access-76gjz\") pod \"payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.520623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.520597 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:46.812253 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.812218 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v"] Oct 31 12:51:46.815186 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:46.815158 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e00f41_6042_44e5_b33e_f9373f5d1c95.slice/crio-f9b7c025d2888f5c7e59874299f07d52b50f2f8be874f5ce3245bb10c00ee216 WatchSource:0}: Error finding container f9b7c025d2888f5c7e59874299f07d52b50f2f8be874f5ce3245bb10c00ee216: Status 404 returned error can't find the container with id f9b7c025d2888f5c7e59874299f07d52b50f2f8be874f5ce3245bb10c00ee216 Oct 31 12:51:46.980852 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.980807 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd"] Oct 31 12:51:46.985661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:46.985619 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.021760 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.019402 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd"] Oct 31 12:51:47.062014 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.061919 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062014 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.061972 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.062079 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.062110 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.062130 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062301 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.062232 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062301 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.062271 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062378 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.062305 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmq5\" (UniqueName: \"kubernetes.io/projected/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-kube-api-access-5lmq5\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.062378 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.062335 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163328 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163294 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163337 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163379 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163410 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163775 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmq5\" (UniqueName: \"kubernetes.io/projected/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-kube-api-access-5lmq5\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163787 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163820 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.163907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.163885 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.164305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.164103 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.164305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.164168 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.164305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.164278 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.165624 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.164734 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.165624 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.164933 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.165624 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.165577 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.167690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.167604 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.168111 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.168088 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.194295 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.194266 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.196886 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.196864 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmq5\" (UniqueName: \"kubernetes.io/projected/8b89ff01-1a45-4ec2-b5bd-75c7178126f2-kube-api-access-5lmq5\") pod \"maas-default-gateway-openshift-default-fc8c56547-vdjwd\" (UID: \"8b89ff01-1a45-4ec2-b5bd-75c7178126f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.300350 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.300317 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:47.621187 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.621108 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" event={"ID":"39e00f41-6042-44e5-b33e-f9373f5d1c95","Type":"ContainerStarted","Data":"f9b7c025d2888f5c7e59874299f07d52b50f2f8be874f5ce3245bb10c00ee216"} Oct 31 12:51:47.638194 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.637512 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd"] Oct 31 12:51:47.641274 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:47.640257 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b89ff01_1a45_4ec2_b5bd_75c7178126f2.slice/crio-afb751719118f66f463d50087b816568f7e778abf39e23d5a940eadf7ad101de WatchSource:0}: Error finding container afb751719118f66f463d50087b816568f7e778abf39e23d5a940eadf7ad101de: Status 404 returned error can't find the container with id afb751719118f66f463d50087b816568f7e778abf39e23d5a940eadf7ad101de Oct 31 12:51:47.645962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.645912 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:47.646112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.646010 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:47.646112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.646060 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:51:47.652027 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.650972 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz"] Oct 31 12:51:47.657170 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.657147 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:47.666628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.665489 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b03bbf98\"" Oct 31 12:51:47.667002 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.666980 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b03bbf98-dockercfg-9zvz6\"" Oct 31 12:51:47.669544 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.669518 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz"] Oct 31 12:51:47.771066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.771022 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0377fb34-98f0-4e22-9f4e-29287c7970a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:47.771226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.771084 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jmh\" (UniqueName: \"kubernetes.io/projected/0377fb34-98f0-4e22-9f4e-29287c7970a1-kube-api-access-t7jmh\") pod \"payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:47.872461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.872373 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0377fb34-98f0-4e22-9f4e-29287c7970a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:47.872461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.872434 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jmh\" (UniqueName: \"kubernetes.io/projected/0377fb34-98f0-4e22-9f4e-29287c7970a1-kube-api-access-t7jmh\") pod \"payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:47.873161 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.873141 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0377fb34-98f0-4e22-9f4e-29287c7970a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:47.886904 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.886874 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jmh\" (UniqueName: \"kubernetes.io/projected/0377fb34-98f0-4e22-9f4e-29287c7970a1-kube-api-access-t7jmh\") pod \"payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:47.981712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:47.981676 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:48.176410 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.175481 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44"] Oct 31 12:51:48.231555 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.231502 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44"] Oct 31 12:51:48.232018 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.231999 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.255682 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.255655 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d48f454a-dockercfg-fnvqk\"" Oct 31 12:51:48.255835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.255628 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d48f454a\"" Oct 31 12:51:48.317922 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.317872 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz"] Oct 31 12:51:48.321109 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:48.321083 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0377fb34_98f0_4e22_9f4e_29287c7970a1.slice/crio-fcf60a66ab7a51d7272db2c520a92169cc1fec7ba0541b3c7eb9f770102c3a87 WatchSource:0}: Error finding container fcf60a66ab7a51d7272db2c520a92169cc1fec7ba0541b3c7eb9f770102c3a87: Status 404 returned error can't find the container with id fcf60a66ab7a51d7272db2c520a92169cc1fec7ba0541b3c7eb9f770102c3a87 Oct 31 12:51:48.408652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.408230 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d48f454a-5fff859755-nmg44\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.408652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.408300 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g86j\" (UniqueName: \"kubernetes.io/projected/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-kube-api-access-8g86j\") pod \"payload-processing-e2e-ait-d48f454a-5fff859755-nmg44\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.512781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.511886 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g86j\" (UniqueName: \"kubernetes.io/projected/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-kube-api-access-8g86j\") pod \"payload-processing-e2e-ait-d48f454a-5fff859755-nmg44\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.512781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.511990 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d48f454a-5fff859755-nmg44\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.512781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.512735 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-d48f454a-5fff859755-nmg44\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.533169 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.533137 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g86j\" (UniqueName: \"kubernetes.io/projected/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-kube-api-access-8g86j\") pod \"payload-processing-e2e-ait-d48f454a-5fff859755-nmg44\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.564919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.564880 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:48.604045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.603584 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk"] Oct 31 12:51:48.608117 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.608095 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:48.618960 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.618699 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1abc54b4\"" Oct 31 12:51:48.618960 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.618799 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1abc54b4-dockercfg-7hpkv\"" Oct 31 12:51:48.627335 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.627243 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk"] Oct 31 12:51:48.633293 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.633228 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" event={"ID":"0377fb34-98f0-4e22-9f4e-29287c7970a1","Type":"ContainerStarted","Data":"fcf60a66ab7a51d7272db2c520a92169cc1fec7ba0541b3c7eb9f770102c3a87"} Oct 31 12:51:48.637770 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.637738 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk"] Oct 31 12:51:48.645481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.642985 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" event={"ID":"8b89ff01-1a45-4ec2-b5bd-75c7178126f2","Type":"ContainerStarted","Data":"233eb4f35eb485c27c0cf8d8b98a650b93a0801d701767c1c8c12fb155d396ba"} Oct 31 12:51:48.645481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.643023 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" event={"ID":"8b89ff01-1a45-4ec2-b5bd-75c7178126f2","Type":"ContainerStarted","Data":"afb751719118f66f463d50087b816568f7e778abf39e23d5a940eadf7ad101de"} Oct 31 12:51:48.645481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.643115 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:48.655881 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.655837 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk"] Oct 31 12:51:48.703121 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.702923 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podStartSLOduration=2.702904169 podStartE2EDuration="2.702904169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:48.688372058 +0000 UTC m=+1202.866090046" watchObservedRunningTime="2026-10-31 12:51:48.702904169 +0000 UTC m=+1202.880622150" Oct 31 12:51:48.725291 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.713249 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:48.725291 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.713407 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m629\" (UniqueName: \"kubernetes.io/projected/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-kube-api-access-6m629\") pod \"payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:48.814401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.814318 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdnm\" (UniqueName: \"kubernetes.io/projected/7e2ba614-7a90-4d12-9920-10583af4294a-kube-api-access-cwdnm\") pod \"payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:48.814401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.814375 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ba614-7a90-4d12-9920-10583af4294a-plugins-config-volume\") pod \"payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:48.814401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.814396 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m629\" (UniqueName: \"kubernetes.io/projected/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-kube-api-access-6m629\") pod \"payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:48.814655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.814424 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:48.816196 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.816170 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:48.829377 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.829346 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m629\" (UniqueName: \"kubernetes.io/projected/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-kube-api-access-6m629\") pod \"payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:48.915040 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.914999 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ba614-7a90-4d12-9920-10583af4294a-plugins-config-volume\") pod \"payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:48.915922 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.915430 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdnm\" (UniqueName: \"kubernetes.io/projected/7e2ba614-7a90-4d12-9920-10583af4294a-kube-api-access-cwdnm\") pod \"payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:48.916120 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.916099 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ba614-7a90-4d12-9920-10583af4294a-plugins-config-volume\") pod \"payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:48.928461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.928427 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44"] Oct 31 12:51:48.932096 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.932068 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdnm\" (UniqueName: \"kubernetes.io/projected/7e2ba614-7a90-4d12-9920-10583af4294a-kube-api-access-cwdnm\") pod \"payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:48.938567 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:48.938539 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd411f3a0_f6f9_41df_a6f2_8a6e6df4c1c2.slice/crio-a2663f1c9e96342c9b18b1c96f8fde3e272568e4d75362d088b92ac5da3660c0 WatchSource:0}: Error finding container a2663f1c9e96342c9b18b1c96f8fde3e272568e4d75362d088b92ac5da3660c0: Status 404 returned error can't find the container with id a2663f1c9e96342c9b18b1c96f8fde3e272568e4d75362d088b92ac5da3660c0 Oct 31 12:51:48.953975 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:48.953943 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:49.004198 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.003821 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:49.304857 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.304762 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:49.305808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.305458 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:49.305808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.305511 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 12:51:49.332987 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.332054 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk"] Oct 31 12:51:49.335372 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:49.335300 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c33f2ac_7d6c_4e7c_bff3_1c6df9c9d83e.slice/crio-8d0e16a10abd689004face68913657f5182a9c56b309448e11c9c4583f191b13 WatchSource:0}: Error finding container 8d0e16a10abd689004face68913657f5182a9c56b309448e11c9c4583f191b13: Status 404 returned error can't find the container with id 8d0e16a10abd689004face68913657f5182a9c56b309448e11c9c4583f191b13 Oct 31 12:51:49.410071 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.410035 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk"] Oct 31 12:51:49.415898 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:49.415854 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ba614_7a90_4d12_9920_10583af4294a.slice/crio-1423cd44bc2385c0bb85ccf72215c67f096e56a25c5822be90ccba0187ecadac WatchSource:0}: Error finding container 1423cd44bc2385c0bb85ccf72215c67f096e56a25c5822be90ccba0187ecadac: Status 404 returned error can't find the container with id 1423cd44bc2385c0bb85ccf72215c67f096e56a25c5822be90ccba0187ecadac Oct 31 12:51:49.649548 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.649417 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" event={"ID":"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e","Type":"ContainerStarted","Data":"8d0e16a10abd689004face68913657f5182a9c56b309448e11c9c4583f191b13"} Oct 31 12:51:49.651920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.651843 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" event={"ID":"7e2ba614-7a90-4d12-9920-10583af4294a","Type":"ContainerStarted","Data":"1423cd44bc2385c0bb85ccf72215c67f096e56a25c5822be90ccba0187ecadac"} Oct 31 12:51:49.654395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:49.654331 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" event={"ID":"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2","Type":"ContainerStarted","Data":"a2663f1c9e96342c9b18b1c96f8fde3e272568e4d75362d088b92ac5da3660c0"} Oct 31 12:51:50.308668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:50.308123 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:50.308668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:50.308185 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 12:51:51.164326 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:51.164282 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:51:51.302705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:51.302221 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:51.302705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:51.302278 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 12:51:52.306534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:52.306063 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:52.306534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:52.306130 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 12:51:53.301401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.300905 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:53.301401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.300961 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 12:51:53.721972 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.721761 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" event={"ID":"39e00f41-6042-44e5-b33e-f9373f5d1c95","Type":"ContainerStarted","Data":"20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344"} Oct 31 12:51:53.721972 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.721896 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:51:53.725345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.725284 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" event={"ID":"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e","Type":"ContainerStarted","Data":"fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666"} Oct 31 12:51:53.725934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.725906 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:51:53.728697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.728668 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" event={"ID":"7e2ba614-7a90-4d12-9920-10583af4294a","Type":"ContainerStarted","Data":"fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504"} Oct 31 12:51:53.728945 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.728901 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:51:53.743282 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.743232 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" podStartSLOduration=1.5519563299999999 podStartE2EDuration="7.743220692s" totalImagesPullingTime="6.191264362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:53.742786399 +0000 UTC m=+1207.920504384" watchObservedRunningTime="2026-10-31 12:51:53.743220692 +0000 UTC m=+1207.920938675" Oct 31 12:51:53.768802 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.768746 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" podStartSLOduration=1.916048273 podStartE2EDuration="5.768723058s" totalImagesPullingTime="3.852674785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:53.766450725 +0000 UTC m=+1207.944168710" watchObservedRunningTime="2026-10-31 12:51:53.768723058 +0000 UTC m=+1207.946441043" Oct 31 12:51:53.787502 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:53.787452 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" podStartSLOduration=2.062244729 podStartE2EDuration="5.787436534s" totalImagesPullingTime="3.725191805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:53.784670542 +0000 UTC m=+1207.962388525" watchObservedRunningTime="2026-10-31 12:51:53.787436534 +0000 UTC m=+1207.965154519" Oct 31 12:51:54.300981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.300869 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:54.300981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.300939 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 12:51:54.735787 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.735747 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" event={"ID":"0377fb34-98f0-4e22-9f4e-29287c7970a1","Type":"ContainerStarted","Data":"80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18"} Oct 31 12:51:54.736306 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.735870 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:51:54.737978 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.737487 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" event={"ID":"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2","Type":"ContainerStarted","Data":"5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421"} Oct 31 12:51:54.737978 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.737543 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:51:54.757303 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.757239 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" podStartSLOduration=2.110966446 podStartE2EDuration="7.757221294s" totalImagesPullingTime="5.646254848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:54.754234712 +0000 UTC m=+1208.931952700" watchObservedRunningTime="2026-10-31 12:51:54.757221294 +0000 UTC m=+1208.934939280" Oct 31 12:51:54.773484 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:54.773428 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" podStartSLOduration=1.635372703 podStartE2EDuration="6.773413499s" totalImagesPullingTime="5.138040796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:54.77186838 +0000 UTC m=+1208.949586365" watchObservedRunningTime="2026-10-31 12:51:54.773413499 +0000 UTC m=+1208.951131511" Oct 31 12:51:55.301434 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:55.301388 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 12:51:55.301677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:55.301455 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 12:51:55.447071 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:55.447035 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:51:55.987665 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:51:55.986624 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:51:56.369304 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.365081 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:56.369304 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.366409 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:56.369304 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.367590 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" Oct 31 12:51:56.490904 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.490867 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp"] Oct 31 12:51:56.495456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.495432 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv"] Oct 31 12:51:56.495594 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.495536 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.502743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.502710 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-785c04b9\"" Oct 31 12:51:56.503590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.503567 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:56.506176 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.506035 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-785c04b9-dockercfg-nrxmb\"" Oct 31 12:51:56.517530 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.517496 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp"] Oct 31 12:51:56.520124 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.520097 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv"] Oct 31 12:51:56.565038 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.564998 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30279950-6bdb-41ed-b712-c2fa9e2473b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:56.565210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.565063 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhrv\" (UniqueName: \"kubernetes.io/projected/30279950-6bdb-41ed-b712-c2fa9e2473b6-kube-api-access-8mhrv\") pod \"payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:56.565210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.565136 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmv6\" (UniqueName: \"kubernetes.io/projected/bc39df69-69dd-4993-a31d-47938f9e5cf5-kube-api-access-wtmv6\") pod \"payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.565210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.565165 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39df69-69dd-4993-a31d-47938f9e5cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.666463 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.666381 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30279950-6bdb-41ed-b712-c2fa9e2473b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:56.666463 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.666440 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhrv\" (UniqueName: \"kubernetes.io/projected/30279950-6bdb-41ed-b712-c2fa9e2473b6-kube-api-access-8mhrv\") pod \"payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:56.666753 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.666470 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmv6\" (UniqueName: \"kubernetes.io/projected/bc39df69-69dd-4993-a31d-47938f9e5cf5-kube-api-access-wtmv6\") pod \"payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.666753 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.666494 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39df69-69dd-4993-a31d-47938f9e5cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.667088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.667066 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30279950-6bdb-41ed-b712-c2fa9e2473b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:56.667176 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.667158 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39df69-69dd-4993-a31d-47938f9e5cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.681189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.681158 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhrv\" (UniqueName: \"kubernetes.io/projected/30279950-6bdb-41ed-b712-c2fa9e2473b6-kube-api-access-8mhrv\") pod \"payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:56.681357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.681336 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmv6\" (UniqueName: \"kubernetes.io/projected/bc39df69-69dd-4993-a31d-47938f9e5cf5-kube-api-access-wtmv6\") pod \"payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.812980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.812938 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:56.839360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:56.839332 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:57.098793 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.098733 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp"] Oct 31 12:51:57.101412 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:57.101381 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc39df69_69dd_4993_a31d_47938f9e5cf5.slice/crio-8957a5b09a7e1bd767e8ce5b84b408538058133a36cf08a1d76ec19425ea8144 WatchSource:0}: Error finding container 8957a5b09a7e1bd767e8ce5b84b408538058133a36cf08a1d76ec19425ea8144: Status 404 returned error can't find the container with id 8957a5b09a7e1bd767e8ce5b84b408538058133a36cf08a1d76ec19425ea8144 Oct 31 12:51:57.141331 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.141299 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv"] Oct 31 12:51:57.146786 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:51:57.146752 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30279950_6bdb_41ed_b712_c2fa9e2473b6.slice/crio-f421c9db8b8c6798d4a28b2ddb97ed3ecde7cf530d63f8b92ca4017136221fde WatchSource:0}: Error finding container f421c9db8b8c6798d4a28b2ddb97ed3ecde7cf530d63f8b92ca4017136221fde: Status 404 returned error can't find the container with id f421c9db8b8c6798d4a28b2ddb97ed3ecde7cf530d63f8b92ca4017136221fde Oct 31 12:51:57.755701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.755658 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" event={"ID":"30279950-6bdb-41ed-b712-c2fa9e2473b6","Type":"ContainerStarted","Data":"fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7"} Oct 31 12:51:57.755912 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.755818 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:51:57.755912 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.755840 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" event={"ID":"30279950-6bdb-41ed-b712-c2fa9e2473b6","Type":"ContainerStarted","Data":"f421c9db8b8c6798d4a28b2ddb97ed3ecde7cf530d63f8b92ca4017136221fde"} Oct 31 12:51:57.757191 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.757162 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" event={"ID":"bc39df69-69dd-4993-a31d-47938f9e5cf5","Type":"ContainerStarted","Data":"d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e"} Oct 31 12:51:57.757321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.757207 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" event={"ID":"bc39df69-69dd-4993-a31d-47938f9e5cf5","Type":"ContainerStarted","Data":"8957a5b09a7e1bd767e8ce5b84b408538058133a36cf08a1d76ec19425ea8144"} Oct 31 12:51:57.757321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.757225 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:51:57.781203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.781147 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" podStartSLOduration=1.781132234 podStartE2EDuration="1.781132234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:57.778077986 +0000 UTC m=+1211.955795969" watchObservedRunningTime="2026-10-31 12:51:57.781132234 +0000 UTC m=+1211.958850217" Oct 31 12:51:57.806991 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:51:57.806937 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" podStartSLOduration=1.8069193829999999 podStartE2EDuration="1.806919383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:51:57.805666352 +0000 UTC m=+1211.983384338" watchObservedRunningTime="2026-10-31 12:51:57.806919383 +0000 UTC m=+1211.984637367" Oct 31 12:52:00.704392 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.704352 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq"] Oct 31 12:52:00.708234 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.708215 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.720574 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.720550 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:52:00.720885 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.720859 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:52:00.721808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.721775 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq"] Oct 31 12:52:00.730057 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.730026 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:52:00.730057 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.730030 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:52:00.807908 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.807865 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.808094 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.808041 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgjq\" (UniqueName: \"kubernetes.io/projected/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-kube-api-access-wsgjq\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.808228 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.808156 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.909283 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.909246 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.909465 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.909318 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgjq\" (UniqueName: \"kubernetes.io/projected/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-kube-api-access-wsgjq\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.909465 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.909357 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.910001 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.909974 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.911604 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.911584 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:00.923061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:00.923034 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgjq\" (UniqueName: \"kubernetes.io/projected/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-kube-api-access-wsgjq\") pod \"maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:01.021805 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:01.021771 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:01.313277 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:01.313185 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq"] Oct 31 12:52:01.315571 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:01.315532 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fe68cd_ae6d_4e2a_92f3_e79ca4ee42d7.slice/crio-82f642bb86b6d569fd6f43cff8ed9b33cd3aad84cbe11f20c5e6f8b0ea80aa0e WatchSource:0}: Error finding container 82f642bb86b6d569fd6f43cff8ed9b33cd3aad84cbe11f20c5e6f8b0ea80aa0e: Status 404 returned error can't find the container with id 82f642bb86b6d569fd6f43cff8ed9b33cd3aad84cbe11f20c5e6f8b0ea80aa0e Oct 31 12:52:01.779100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:01.779066 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" event={"ID":"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7","Type":"ContainerStarted","Data":"82f642bb86b6d569fd6f43cff8ed9b33cd3aad84cbe11f20c5e6f8b0ea80aa0e"} Oct 31 12:52:04.740814 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:04.740784 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:52:04.741225 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:04.740863 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:52:04.805907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:04.805868 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" event={"ID":"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7","Type":"ContainerStarted","Data":"7c7f2a7341d494f5743c77e20aa90c15e990bb2dbdda2b52e3a59e94e9aedc51"} Oct 31 12:52:04.843166 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:04.843115 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" podStartSLOduration=2.02714797 podStartE2EDuration="4.84309996s" totalImagesPullingTime="2.81595199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:04.8424987 +0000 UTC m=+1219.020216691" watchObservedRunningTime="2026-10-31 12:52:04.84309996 +0000 UTC m=+1219.020817943" Oct 31 12:52:04.994400 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:04.994314 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:05.747727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:05.747694 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:52:05.748164 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:05.747762 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:52:05.748164 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:05.747794 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:52:05.816028 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:05.815996 5074 generic.go:356] "Generic (PLEG): container finished" podID="b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" containerID="7c7f2a7341d494f5743c77e20aa90c15e990bb2dbdda2b52e3a59e94e9aedc51" exitCode=0 Oct 31 12:52:05.816204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:05.816084 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" event={"ID":"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7","Type":"ContainerDied","Data":"7c7f2a7341d494f5743c77e20aa90c15e990bb2dbdda2b52e3a59e94e9aedc51"} Oct 31 12:52:06.136902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.136819 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz"] Oct 31 12:52:06.140800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.140772 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.150091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.150063 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz"] Oct 31 12:52:06.263008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.262969 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsx4d\" (UniqueName: \"kubernetes.io/projected/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-kube-api-access-gsx4d\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.263180 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.263042 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.263180 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.263101 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.364686 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.364654 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsx4d\" (UniqueName: \"kubernetes.io/projected/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-kube-api-access-gsx4d\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.364860 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.364717 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.364860 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.364738 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.365664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.365608 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.367013 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.366990 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.377421 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.377389 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsx4d\" (UniqueName: \"kubernetes.io/projected/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-kube-api-access-gsx4d\") pod \"maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.455025 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.454942 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:06.754018 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.753991 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz"] Oct 31 12:52:06.755819 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:06.755790 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe7cdf7_cfc7_482b_ad43_2ae8d09e836f.slice/crio-4fffda4fe0f36c46b9783d78308e91f082ae2448f0379690381aec04b95b58f4 WatchSource:0}: Error finding container 4fffda4fe0f36c46b9783d78308e91f082ae2448f0379690381aec04b95b58f4: Status 404 returned error can't find the container with id 4fffda4fe0f36c46b9783d78308e91f082ae2448f0379690381aec04b95b58f4 Oct 31 12:52:06.822457 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.822423 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" event={"ID":"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f","Type":"ContainerStarted","Data":"4fffda4fe0f36c46b9783d78308e91f082ae2448f0379690381aec04b95b58f4"} Oct 31 12:52:06.939688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:06.939662 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:07.069809 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.069779 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-xks-ca\") pod \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " Oct 31 12:52:07.069981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.069828 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-kube-api-access-wsgjq\" (UniqueName: \"kubernetes.io/projected/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-kube-api-access-wsgjq\") pod \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " Oct 31 12:52:07.069981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.069855 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-openshift-ca\") pod \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\" (UID: \"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7\") " Oct 31 12:52:07.070536 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.070498 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-openshift-ca" pod "b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" (UID: "b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:07.072220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.072194 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-kube-api-access-wsgjq" pod "b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" (UID: "b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7"). InnerVolumeSpecName "kube-api-access-wsgjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:07.072404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.072383 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-xks-ca" pod "b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" (UID: "b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:07.171221 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.171193 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:07.171221 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.171220 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgjq\" (UniqueName: \"kubernetes.io/projected/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-kube-api-access-wsgjq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:07.171221 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.171234 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:07.829986 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.829947 5074 generic.go:356] "Generic (PLEG): container finished" podID="efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" containerID="0ff5873f70665bf8de14c8c5ff0ab14537d14b22f360edb4396e2009115f1b90" exitCode=0 Oct 31 12:52:07.830409 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.830035 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" event={"ID":"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f","Type":"ContainerDied","Data":"0ff5873f70665bf8de14c8c5ff0ab14537d14b22f360edb4396e2009115f1b90"} Oct 31 12:52:07.831391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.831372 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" Oct 31 12:52:07.831488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.831409 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq" event={"ID":"b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7","Type":"ContainerDied","Data":"82f642bb86b6d569fd6f43cff8ed9b33cd3aad84cbe11f20c5e6f8b0ea80aa0e"} Oct 31 12:52:07.831488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:07.831444 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82f642bb86b6d569fd6f43cff8ed9b33cd3aad84cbe11f20c5e6f8b0ea80aa0e" Oct 31 12:52:08.766414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:08.766383 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:52:08.766581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:08.766459 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:52:08.980077 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:08.980051 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:09.091555 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.091471 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-openshift-ca\") pod \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " Oct 31 12:52:09.091555 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.091528 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-xks-ca\") pod \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " Oct 31 12:52:09.091791 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.091561 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-kube-api-access-gsx4d\" (UniqueName: \"kubernetes.io/projected/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-kube-api-access-gsx4d\") pod \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\" (UID: \"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f\") " Oct 31 12:52:09.091958 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.091927 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-openshift-ca" pod "efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" (UID: "efe7cdf7-cfc7-482b-ad43-2ae8d09e836f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:09.093696 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.093670 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-xks-ca" pod "efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" (UID: "efe7cdf7-cfc7-482b-ad43-2ae8d09e836f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:09.093794 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.093771 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-kube-api-access-gsx4d" pod "efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" (UID: "efe7cdf7-cfc7-482b-ad43-2ae8d09e836f"). InnerVolumeSpecName "kube-api-access-gsx4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:09.192542 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.192510 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsx4d\" (UniqueName: \"kubernetes.io/projected/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-kube-api-access-gsx4d\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:09.192542 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.192538 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:09.192542 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.192550 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:09.844619 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.844587 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" Oct 31 12:52:09.844818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.844592 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz" event={"ID":"efe7cdf7-cfc7-482b-ad43-2ae8d09e836f","Type":"ContainerDied","Data":"4fffda4fe0f36c46b9783d78308e91f082ae2448f0379690381aec04b95b58f4"} Oct 31 12:52:09.844818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:09.844723 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fffda4fe0f36c46b9783d78308e91f082ae2448f0379690381aec04b95b58f4" Oct 31 12:52:10.881792 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:10.881757 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v"] Oct 31 12:52:11.889895 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:11.889860 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq"] Oct 31 12:52:11.918746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:11.918708 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cf7e6604-8gbmq"] Oct 31 12:52:11.937537 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:11.937506 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" path="/var/lib/kubelet/pods/b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7/volumes" Oct 31 12:52:12.749192 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.749154 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9"] Oct 31 12:52:12.749669 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.749654 5074 cpu_manager.go:414] "containerMap: removing container" podUID="efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" containerName="revoke-keys" Oct 31 12:52:12.749739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.749670 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" containerName="revoke-keys" Oct 31 12:52:12.749739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.749687 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" containerName="revoke-keys" Oct 31 12:52:12.749739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.749694 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" containerName="revoke-keys" Oct 31 12:52:12.749894 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.749777 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" containerName="revoke-keys" Oct 31 12:52:12.749894 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.749792 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fe68cd-ae6d-4e2a-92f3-e79ca4ee42d7" containerName="revoke-keys" Oct 31 12:52:12.753358 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.753339 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.760715 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.760688 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:52:12.762148 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.762033 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:52:12.762148 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.762124 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:52:12.762436 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.762418 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:52:12.765414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.765387 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9"] Oct 31 12:52:12.829624 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.829590 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.829797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.829755 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghhq\" (UniqueName: \"kubernetes.io/projected/f20e6b12-3872-4cbc-a20f-c0a02426b17e-kube-api-access-7ghhq\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.829841 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.829812 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.931185 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.931147 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghhq\" (UniqueName: \"kubernetes.io/projected/f20e6b12-3872-4cbc-a20f-c0a02426b17e-kube-api-access-7ghhq\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.931668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.931202 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.931668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.931227 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.931882 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.931861 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.933583 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.933554 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:12.944970 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:12.944939 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghhq\" (UniqueName: \"kubernetes.io/projected/f20e6b12-3872-4cbc-a20f-c0a02426b17e-kube-api-access-7ghhq\") pod \"maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:13.067614 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.067534 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:13.357972 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.357925 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9"] Oct 31 12:52:13.380320 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.380284 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf"] Oct 31 12:52:13.391621 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.390510 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.397491 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.397455 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf"] Oct 31 12:52:13.399601 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.399581 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-47bcd9-openshift-default-dockercfg-dwvfp\"" Oct 31 12:52:13.539160 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.538884 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539160 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.538990 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsj2q\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-kube-api-access-hsj2q\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539160 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.539040 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539160 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.539110 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.539184 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.539262 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.539374 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.539458 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230e0627-e62e-4f58-ac8e-91dc89cf2319-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.539757 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.539565 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.640913 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.640816 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.640913 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.640900 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.640946 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsj2q\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-kube-api-access-hsj2q\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.640978 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641026 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641049 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641328 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641228 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641328 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641270 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641428 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641325 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230e0627-e62e-4f58-ac8e-91dc89cf2319-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641428 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641374 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641428 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641408 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641614 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641591 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641857 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641832 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.641988 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.641971 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230e0627-e62e-4f58-ac8e-91dc89cf2319-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.643837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.643808 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.644002 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.643985 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.654951 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.654921 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsj2q\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-kube-api-access-hsj2q\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.656024 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.656003 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.725824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.725793 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:13.873488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.873445 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" event={"ID":"f20e6b12-3872-4cbc-a20f-c0a02426b17e","Type":"ContainerStarted","Data":"952e9d6a1df13205ccafbff2048708a873a905d9730ec0a988591b1361da03d6"} Oct 31 12:52:13.873673 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.873555 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" event={"ID":"f20e6b12-3872-4cbc-a20f-c0a02426b17e","Type":"ContainerStarted","Data":"a5c35b63724e6b7746a056029cd9611f4d5184af8089421a6b1501acde906932"} Oct 31 12:52:13.925683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.925533 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" podStartSLOduration=1.925512741 podStartE2EDuration="1.925512741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:13.924929698 +0000 UTC m=+1228.102647683" watchObservedRunningTime="2026-10-31 12:52:13.925512741 +0000 UTC m=+1228.103230727" Oct 31 12:52:13.989113 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:13.989079 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:14.073211 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.073181 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf"] Oct 31 12:52:14.073695 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:14.073659 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230e0627_e62e_4f58_ac8e_91dc89cf2319.slice/crio-c19cda3e8082a6d8b531e0d89cd6264f6c2cd0f7de646da1c7abc506205f3e4c WatchSource:0}: Error finding container c19cda3e8082a6d8b531e0d89cd6264f6c2cd0f7de646da1c7abc506205f3e4c: Status 404 returned error can't find the container with id c19cda3e8082a6d8b531e0d89cd6264f6c2cd0f7de646da1c7abc506205f3e4c Oct 31 12:52:14.076171 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.076140 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:14.076242 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.076203 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:14.076242 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.076231 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:14.880627 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.880586 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" event={"ID":"230e0627-e62e-4f58-ac8e-91dc89cf2319","Type":"ContainerStarted","Data":"aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f"} Oct 31 12:52:14.880863 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.880664 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" event={"ID":"230e0627-e62e-4f58-ac8e-91dc89cf2319","Type":"ContainerStarted","Data":"c19cda3e8082a6d8b531e0d89cd6264f6c2cd0f7de646da1c7abc506205f3e4c"} Oct 31 12:52:14.882303 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.882271 5074 generic.go:356] "Generic (PLEG): container finished" podID="f20e6b12-3872-4cbc-a20f-c0a02426b17e" containerID="952e9d6a1df13205ccafbff2048708a873a905d9730ec0a988591b1361da03d6" exitCode=0 Oct 31 12:52:14.882450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.882347 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" event={"ID":"f20e6b12-3872-4cbc-a20f-c0a02426b17e","Type":"ContainerDied","Data":"952e9d6a1df13205ccafbff2048708a873a905d9730ec0a988591b1361da03d6"} Oct 31 12:52:14.905440 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:14.905380 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podStartSLOduration=1.9053671269999999 podStartE2EDuration="1.905367127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:14.903959671 +0000 UTC m=+1229.081677666" watchObservedRunningTime="2026-10-31 12:52:14.905367127 +0000 UTC m=+1229.083085111" Oct 31 12:52:15.726945 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:15.726904 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:15.732284 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:15.732260 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:15.882391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:15.882329 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" podUID="39e00f41-6042-44e5-b33e-f9373f5d1c95" containerName="payload-pre-processing" containerID="cri-o://20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344" gracePeriod=25 Oct 31 12:52:15.889195 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:15.889064 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:15.890252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:15.890231 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:16.058616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.058285 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:16.161061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.161027 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:52:16.169750 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.169724 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-openshift-ca\") pod \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " Oct 31 12:52:16.169887 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.169862 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20e6b12-3872-4cbc-a20f-c0a02426b17e-kube-api-access-7ghhq\" (UniqueName: \"kubernetes.io/projected/f20e6b12-3872-4cbc-a20f-c0a02426b17e-kube-api-access-7ghhq\") pod \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " Oct 31 12:52:16.169936 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.169916 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-xks-ca\") pod \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\" (UID: \"f20e6b12-3872-4cbc-a20f-c0a02426b17e\") " Oct 31 12:52:16.170121 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.170098 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-openshift-ca" pod "f20e6b12-3872-4cbc-a20f-c0a02426b17e" (UID: "f20e6b12-3872-4cbc-a20f-c0a02426b17e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:16.170244 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.170229 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:16.171880 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.171850 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-xks-ca" pod "f20e6b12-3872-4cbc-a20f-c0a02426b17e" (UID: "f20e6b12-3872-4cbc-a20f-c0a02426b17e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:16.172027 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.172006 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20e6b12-3872-4cbc-a20f-c0a02426b17e-kube-api-access-7ghhq" pod "f20e6b12-3872-4cbc-a20f-c0a02426b17e" (UID: "f20e6b12-3872-4cbc-a20f-c0a02426b17e"). InnerVolumeSpecName "kube-api-access-7ghhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:16.270659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.270543 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e00f41-6042-44e5-b33e-f9373f5d1c95-kube-api-access-76gjz\" (UniqueName: \"kubernetes.io/projected/39e00f41-6042-44e5-b33e-f9373f5d1c95-kube-api-access-76gjz\") pod \"39e00f41-6042-44e5-b33e-f9373f5d1c95\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " Oct 31 12:52:16.270835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.270712 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e00f41-6042-44e5-b33e-f9373f5d1c95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e00f41-6042-44e5-b33e-f9373f5d1c95-plugins-config-volume\") pod \"39e00f41-6042-44e5-b33e-f9373f5d1c95\" (UID: \"39e00f41-6042-44e5-b33e-f9373f5d1c95\") " Oct 31 12:52:16.271111 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.271085 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ghhq\" (UniqueName: \"kubernetes.io/projected/f20e6b12-3872-4cbc-a20f-c0a02426b17e-kube-api-access-7ghhq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:16.271215 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.271201 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20e6b12-3872-4cbc-a20f-c0a02426b17e-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:16.271282 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.271147 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e00f41-6042-44e5-b33e-f9373f5d1c95-plugins-config-volume" pod "39e00f41-6042-44e5-b33e-f9373f5d1c95" (UID: "39e00f41-6042-44e5-b33e-f9373f5d1c95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:16.272963 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.272943 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e00f41-6042-44e5-b33e-f9373f5d1c95-kube-api-access-76gjz" pod "39e00f41-6042-44e5-b33e-f9373f5d1c95" (UID: "39e00f41-6042-44e5-b33e-f9373f5d1c95"). InnerVolumeSpecName "kube-api-access-76gjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:16.291117 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.291068 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz"] Oct 31 12:52:16.372123 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.372095 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gjz\" (UniqueName: \"kubernetes.io/projected/39e00f41-6042-44e5-b33e-f9373f5d1c95-kube-api-access-76gjz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:16.372123 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.372124 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e00f41-6042-44e5-b33e-f9373f5d1c95-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:16.895934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.895893 5074 generic.go:356] "Generic (PLEG): container finished" podID="39e00f41-6042-44e5-b33e-f9373f5d1c95" containerID="20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344" exitCode=0 Oct 31 12:52:16.896456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.895964 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" Oct 31 12:52:16.896456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.895989 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" event={"ID":"39e00f41-6042-44e5-b33e-f9373f5d1c95","Type":"ContainerDied","Data":"20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344"} Oct 31 12:52:16.896456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.896089 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v" event={"ID":"39e00f41-6042-44e5-b33e-f9373f5d1c95","Type":"ContainerDied","Data":"f9b7c025d2888f5c7e59874299f07d52b50f2f8be874f5ce3245bb10c00ee216"} Oct 31 12:52:16.896456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.896124 5074 scope.go:122] "RemoveContainer" containerID="20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344" Oct 31 12:52:16.898238 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.898217 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" Oct 31 12:52:16.898359 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.898238 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9" event={"ID":"f20e6b12-3872-4cbc-a20f-c0a02426b17e","Type":"ContainerDied","Data":"a5c35b63724e6b7746a056029cd9611f4d5184af8089421a6b1501acde906932"} Oct 31 12:52:16.898359 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.898284 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5c35b63724e6b7746a056029cd9611f4d5184af8089421a6b1501acde906932" Oct 31 12:52:16.909717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.909697 5074 scope.go:122] "RemoveContainer" containerID="20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344" Oct 31 12:52:16.910050 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:16.910028 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344\": container with ID starting with 20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344 not found: ID does not exist" containerID="20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344" Oct 31 12:52:16.910102 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.910060 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344"} err="failed to get container status \"20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344\": rpc error: code = NotFound desc = could not find container \"20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344\": container with ID starting with 20f704930fd01c21d48508424603ee0ff9c8372ee643c3b8cf1ef48880b41344 not found: ID does not exist" Oct 31 12:52:16.959390 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.959357 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v"] Oct 31 12:52:16.974347 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:16.974314 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf7e6604-c5d54df5f-pdc6v"] Oct 31 12:52:17.160614 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.160544 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6"] Oct 31 12:52:17.161045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.161031 5074 cpu_manager.go:414] "containerMap: removing container" podUID="39e00f41-6042-44e5-b33e-f9373f5d1c95" containerName="payload-pre-processing" Oct 31 12:52:17.161114 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.161047 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e00f41-6042-44e5-b33e-f9373f5d1c95" containerName="payload-pre-processing" Oct 31 12:52:17.161114 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.161061 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f20e6b12-3872-4cbc-a20f-c0a02426b17e" containerName="revoke-keys" Oct 31 12:52:17.161114 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.161067 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20e6b12-3872-4cbc-a20f-c0a02426b17e" containerName="revoke-keys" Oct 31 12:52:17.161204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.161140 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e00f41-6042-44e5-b33e-f9373f5d1c95" containerName="payload-pre-processing" Oct 31 12:52:17.161204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.161150 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20e6b12-3872-4cbc-a20f-c0a02426b17e" containerName="revoke-keys" Oct 31 12:52:17.165743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.165727 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.171826 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.171802 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:52:17.171959 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.171949 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:52:17.172052 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.172033 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:52:17.172262 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.172244 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:52:17.245178 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.245139 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6"] Oct 31 12:52:17.294015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.293983 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.294204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.294058 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5vt\" (UniqueName: \"kubernetes.io/projected/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-kube-api-access-dx5vt\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.294204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.294163 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.395443 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.395403 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5vt\" (UniqueName: \"kubernetes.io/projected/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-kube-api-access-dx5vt\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.395443 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.395453 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.395753 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.395533 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.396391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.396348 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.398964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.398936 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.414487 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.414422 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5vt\" (UniqueName: \"kubernetes.io/projected/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-kube-api-access-dx5vt\") pod \"maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.480416 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.480379 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:17.786779 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.786746 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6"] Oct 31 12:52:17.786963 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:17.786938 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dcca93_c9ba_4ba8_87c1_17cab7e044dd.slice/crio-6ed9f0f27a3047ceffc663f2962e006ccbb5b77ccb78437590de782dc9c41c03 WatchSource:0}: Error finding container 6ed9f0f27a3047ceffc663f2962e006ccbb5b77ccb78437590de782dc9c41c03: Status 404 returned error can't find the container with id 6ed9f0f27a3047ceffc663f2962e006ccbb5b77ccb78437590de782dc9c41c03 Oct 31 12:52:17.905698 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.905654 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" event={"ID":"22dcca93-c9ba-4ba8-87c1-17cab7e044dd","Type":"ContainerStarted","Data":"6ed9f0f27a3047ceffc663f2962e006ccbb5b77ccb78437590de782dc9c41c03"} Oct 31 12:52:17.937352 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:17.937320 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e00f41-6042-44e5-b33e-f9373f5d1c95" path="/var/lib/kubelet/pods/39e00f41-6042-44e5-b33e-f9373f5d1c95/volumes" Oct 31 12:52:18.788120 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:18.788082 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz"] Oct 31 12:52:18.796601 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:18.796570 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b03bbf98-zv6tz"] Oct 31 12:52:18.916004 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:18.915970 5074 generic.go:356] "Generic (PLEG): container finished" podID="22dcca93-c9ba-4ba8-87c1-17cab7e044dd" containerID="9c72c143db34ee2ce34b55256df6b2f55fd0e6a55c153e724427303fba0d3d0a" exitCode=0 Oct 31 12:52:18.916405 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:18.916061 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" event={"ID":"22dcca93-c9ba-4ba8-87c1-17cab7e044dd","Type":"ContainerDied","Data":"9c72c143db34ee2ce34b55256df6b2f55fd0e6a55c153e724427303fba0d3d0a"} Oct 31 12:52:19.937702 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:19.937672 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe7cdf7-cfc7-482b-ad43-2ae8d09e836f" path="/var/lib/kubelet/pods/efe7cdf7-cfc7-482b-ad43-2ae8d09e836f/volumes" Oct 31 12:52:19.971579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:19.971545 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b"] Oct 31 12:52:19.976675 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:19.976623 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:19.995560 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:19.995526 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b"] Oct 31 12:52:20.016625 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.016588 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhs5f\" (UniqueName: \"kubernetes.io/projected/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-kube-api-access-nhs5f\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.016809 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.016659 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.016809 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.016758 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.071265 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.071238 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:20.117611 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.117574 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-openshift-ca\") pod \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " Oct 31 12:52:20.117825 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.117674 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-xks-ca\") pod \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " Oct 31 12:52:20.117825 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.117795 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-kube-api-access-dx5vt\" (UniqueName: \"kubernetes.io/projected/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-kube-api-access-dx5vt\") pod \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\" (UID: \"22dcca93-c9ba-4ba8-87c1-17cab7e044dd\") " Oct 31 12:52:20.117954 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.117922 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.118030 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.118000 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-openshift-ca" pod "22dcca93-c9ba-4ba8-87c1-17cab7e044dd" (UID: "22dcca93-c9ba-4ba8-87c1-17cab7e044dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:20.118116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.118096 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhs5f\" (UniqueName: \"kubernetes.io/projected/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-kube-api-access-nhs5f\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.118175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.118138 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.118251 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.118234 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:20.118972 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.118946 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.120159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.120135 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-kube-api-access-dx5vt" pod "22dcca93-c9ba-4ba8-87c1-17cab7e044dd" (UID: "22dcca93-c9ba-4ba8-87c1-17cab7e044dd"). InnerVolumeSpecName "kube-api-access-dx5vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:20.120693 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.120672 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.121048 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.121024 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-xks-ca" pod "22dcca93-c9ba-4ba8-87c1-17cab7e044dd" (UID: "22dcca93-c9ba-4ba8-87c1-17cab7e044dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:20.130824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.130804 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhs5f\" (UniqueName: \"kubernetes.io/projected/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-kube-api-access-nhs5f\") pod \"maas-api-revoke-keys-e2e-derive-785c04b9-f877b\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.181984 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.181871 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:20.219073 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.219036 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:20.219073 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.219062 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5vt\" (UniqueName: \"kubernetes.io/projected/22dcca93-c9ba-4ba8-87c1-17cab7e044dd-kube-api-access-dx5vt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:20.307810 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.307771 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:20.597604 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.597571 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b"] Oct 31 12:52:20.599848 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:20.599814 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3cc385_d3ae_43e2_aa72_fffec8dc725f.slice/crio-a769a4358a8d9111a9297cf72d27bba2bad30170d325dd7ca2d3f897f411095a WatchSource:0}: Error finding container a769a4358a8d9111a9297cf72d27bba2bad30170d325dd7ca2d3f897f411095a: Status 404 returned error can't find the container with id a769a4358a8d9111a9297cf72d27bba2bad30170d325dd7ca2d3f897f411095a Oct 31 12:52:20.927514 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.927478 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" event={"ID":"22dcca93-c9ba-4ba8-87c1-17cab7e044dd","Type":"ContainerDied","Data":"6ed9f0f27a3047ceffc663f2962e006ccbb5b77ccb78437590de782dc9c41c03"} Oct 31 12:52:20.927679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.927530 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed9f0f27a3047ceffc663f2962e006ccbb5b77ccb78437590de782dc9c41c03" Oct 31 12:52:20.927679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.927597 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6" Oct 31 12:52:20.931902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.931876 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" event={"ID":"2e3cc385-d3ae-43e2-aa72-fffec8dc725f","Type":"ContainerStarted","Data":"d4c30a4e83c831e13a3583bb5f5da2aff71e51783374f8c1336e5e52806b368f"} Oct 31 12:52:20.932023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.931923 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" event={"ID":"2e3cc385-d3ae-43e2-aa72-fffec8dc725f","Type":"ContainerStarted","Data":"a769a4358a8d9111a9297cf72d27bba2bad30170d325dd7ca2d3f897f411095a"} Oct 31 12:52:20.960815 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:20.960768 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" podStartSLOduration=1.960755595 podStartE2EDuration="1.960755595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:20.958014862 +0000 UTC m=+1235.135732849" watchObservedRunningTime="2026-10-31 12:52:20.960755595 +0000 UTC m=+1235.138473608" Oct 31 12:52:21.291694 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.291624 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" podUID="0377fb34-98f0-4e22-9f4e-29287c7970a1" containerName="payload-pre-processing" containerID="cri-o://80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18" gracePeriod=25 Oct 31 12:52:21.568020 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.567987 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:52:21.634970 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.634935 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0377fb34-98f0-4e22-9f4e-29287c7970a1-kube-api-access-t7jmh\" (UniqueName: \"kubernetes.io/projected/0377fb34-98f0-4e22-9f4e-29287c7970a1-kube-api-access-t7jmh\") pod \"0377fb34-98f0-4e22-9f4e-29287c7970a1\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " Oct 31 12:52:21.635144 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.635079 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0377fb34-98f0-4e22-9f4e-29287c7970a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0377fb34-98f0-4e22-9f4e-29287c7970a1-plugins-config-volume\") pod \"0377fb34-98f0-4e22-9f4e-29287c7970a1\" (UID: \"0377fb34-98f0-4e22-9f4e-29287c7970a1\") " Oct 31 12:52:21.635549 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.635521 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0377fb34-98f0-4e22-9f4e-29287c7970a1-plugins-config-volume" pod "0377fb34-98f0-4e22-9f4e-29287c7970a1" (UID: "0377fb34-98f0-4e22-9f4e-29287c7970a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:21.637151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.637127 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0377fb34-98f0-4e22-9f4e-29287c7970a1-kube-api-access-t7jmh" pod "0377fb34-98f0-4e22-9f4e-29287c7970a1" (UID: "0377fb34-98f0-4e22-9f4e-29287c7970a1"). InnerVolumeSpecName "kube-api-access-t7jmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:21.736073 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.736036 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0377fb34-98f0-4e22-9f4e-29287c7970a1-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:21.736073 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.736067 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7jmh\" (UniqueName: \"kubernetes.io/projected/0377fb34-98f0-4e22-9f4e-29287c7970a1-kube-api-access-t7jmh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:21.943964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.943865 5074 generic.go:356] "Generic (PLEG): container finished" podID="0377fb34-98f0-4e22-9f4e-29287c7970a1" containerID="80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18" exitCode=0 Oct 31 12:52:21.943964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.943926 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" Oct 31 12:52:21.944175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.944043 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" event={"ID":"0377fb34-98f0-4e22-9f4e-29287c7970a1","Type":"ContainerDied","Data":"80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18"} Oct 31 12:52:21.944175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.944112 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz" event={"ID":"0377fb34-98f0-4e22-9f4e-29287c7970a1","Type":"ContainerDied","Data":"fcf60a66ab7a51d7272db2c520a92169cc1fec7ba0541b3c7eb9f770102c3a87"} Oct 31 12:52:21.944175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.944144 5074 scope.go:122] "RemoveContainer" containerID="80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18" Oct 31 12:52:21.946845 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.946817 5074 generic.go:356] "Generic (PLEG): container finished" podID="2e3cc385-d3ae-43e2-aa72-fffec8dc725f" containerID="d4c30a4e83c831e13a3583bb5f5da2aff71e51783374f8c1336e5e52806b368f" exitCode=0 Oct 31 12:52:21.946963 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.946858 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" event={"ID":"2e3cc385-d3ae-43e2-aa72-fffec8dc725f","Type":"ContainerDied","Data":"d4c30a4e83c831e13a3583bb5f5da2aff71e51783374f8c1336e5e52806b368f"} Oct 31 12:52:21.961768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.961739 5074 scope.go:122] "RemoveContainer" containerID="80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18" Oct 31 12:52:21.962158 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:21.962029 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18\": container with ID starting with 80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18 not found: ID does not exist" containerID="80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18" Oct 31 12:52:21.962158 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.962058 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18"} err="failed to get container status \"80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18\": rpc error: code = NotFound desc = could not find container \"80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18\": container with ID starting with 80a8ecfe7bc774b01fbcf9d8dd1f23c320092bd4b8f66e18abec0b1a3748ba18 not found: ID does not exist" Oct 31 12:52:21.986308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.986270 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz"] Oct 31 12:52:21.996472 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:21.996432 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b03bbf98-99646f5-xr7cz"] Oct 31 12:52:22.914517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:22.914482 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk"] Oct 31 12:52:22.920081 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:22.920046 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk"] Oct 31 12:52:23.105085 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.105057 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:23.150966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.150937 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-openshift-ca\") pod \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " Oct 31 12:52:23.151129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.150984 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-kube-api-access-nhs5f\" (UniqueName: \"kubernetes.io/projected/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-kube-api-access-nhs5f\") pod \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " Oct 31 12:52:23.151129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.151054 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-xks-ca\") pod \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\" (UID: \"2e3cc385-d3ae-43e2-aa72-fffec8dc725f\") " Oct 31 12:52:23.151391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.151363 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-openshift-ca" pod "2e3cc385-d3ae-43e2-aa72-fffec8dc725f" (UID: "2e3cc385-d3ae-43e2-aa72-fffec8dc725f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:23.153149 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.153117 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-xks-ca" pod "2e3cc385-d3ae-43e2-aa72-fffec8dc725f" (UID: "2e3cc385-d3ae-43e2-aa72-fffec8dc725f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:52:23.153250 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.153198 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-kube-api-access-nhs5f" pod "2e3cc385-d3ae-43e2-aa72-fffec8dc725f" (UID: "2e3cc385-d3ae-43e2-aa72-fffec8dc725f"). InnerVolumeSpecName "kube-api-access-nhs5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:23.252583 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.252550 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:23.252583 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.252579 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhs5f\" (UniqueName: \"kubernetes.io/projected/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-kube-api-access-nhs5f\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:23.252815 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.252597 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3cc385-d3ae-43e2-aa72-fffec8dc725f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:23.938918 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.938880 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0377fb34-98f0-4e22-9f4e-29287c7970a1" path="/var/lib/kubelet/pods/0377fb34-98f0-4e22-9f4e-29287c7970a1/volumes" Oct 31 12:52:23.960677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.960650 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" Oct 31 12:52:23.960677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.960665 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b" event={"ID":"2e3cc385-d3ae-43e2-aa72-fffec8dc725f","Type":"ContainerDied","Data":"a769a4358a8d9111a9297cf72d27bba2bad30170d325dd7ca2d3f897f411095a"} Oct 31 12:52:23.960837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:23.960714 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a769a4358a8d9111a9297cf72d27bba2bad30170d325dd7ca2d3f897f411095a" Oct 31 12:52:26.003746 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:26.003711 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:52:27.263274 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:27.263232 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44"] Oct 31 12:52:27.558564 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:27.558481 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9"] Oct 31 12:52:27.561452 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:27.561424 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1abc54b4-9nsw9"] Oct 31 12:52:27.915083 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:27.914980 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" podUID="7e2ba614-7a90-4d12-9920-10583af4294a" containerName="payload-processing" containerID="cri-o://fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504" gracePeriod=25 Oct 31 12:52:27.920550 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:27.920516 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" podUID="2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" containerName="payload-pre-processing" containerID="cri-o://fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666" gracePeriod=25 Oct 31 12:52:27.941073 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:27.941041 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20e6b12-3872-4cbc-a20f-c0a02426b17e" path="/var/lib/kubelet/pods/f20e6b12-3872-4cbc-a20f-c0a02426b17e/volumes" Oct 31 12:52:28.278657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.274797 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:52:28.330672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.330090 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ba614-7a90-4d12-9920-10583af4294a-kube-api-access-cwdnm\" (UniqueName: \"kubernetes.io/projected/7e2ba614-7a90-4d12-9920-10583af4294a-kube-api-access-cwdnm\") pod \"7e2ba614-7a90-4d12-9920-10583af4294a\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " Oct 31 12:52:28.330907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.330884 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2ba614-7a90-4d12-9920-10583af4294a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ba614-7a90-4d12-9920-10583af4294a-plugins-config-volume\") pod \"7e2ba614-7a90-4d12-9920-10583af4294a\" (UID: \"7e2ba614-7a90-4d12-9920-10583af4294a\") " Oct 31 12:52:28.331506 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.331476 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2ba614-7a90-4d12-9920-10583af4294a-plugins-config-volume" pod "7e2ba614-7a90-4d12-9920-10583af4294a" (UID: "7e2ba614-7a90-4d12-9920-10583af4294a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:28.356749 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.356673 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ba614-7a90-4d12-9920-10583af4294a-kube-api-access-cwdnm" pod "7e2ba614-7a90-4d12-9920-10583af4294a" (UID: "7e2ba614-7a90-4d12-9920-10583af4294a"). InnerVolumeSpecName "kube-api-access-cwdnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:28.366626 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.366593 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:52:28.432726 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.432693 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-plugins-config-volume\") pod \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " Oct 31 12:52:28.432899 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.432746 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-kube-api-access-6m629\" (UniqueName: \"kubernetes.io/projected/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-kube-api-access-6m629\") pod \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\" (UID: \"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e\") " Oct 31 12:52:28.433121 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.433101 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwdnm\" (UniqueName: \"kubernetes.io/projected/7e2ba614-7a90-4d12-9920-10583af4294a-kube-api-access-cwdnm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:28.433179 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.433129 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ba614-7a90-4d12-9920-10583af4294a-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:28.433179 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.433141 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-plugins-config-volume" pod "2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" (UID: "2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:28.435070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.435039 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-kube-api-access-6m629" pod "2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" (UID: "2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e"). InnerVolumeSpecName "kube-api-access-6m629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:28.534102 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.534016 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:28.534102 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:28.534042 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m629\" (UniqueName: \"kubernetes.io/projected/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e-kube-api-access-6m629\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:29.015869 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.015828 5074 generic.go:356] "Generic (PLEG): container finished" podID="7e2ba614-7a90-4d12-9920-10583af4294a" containerID="fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504" exitCode=0 Oct 31 12:52:29.016065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.015891 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" event={"ID":"7e2ba614-7a90-4d12-9920-10583af4294a","Type":"ContainerDied","Data":"fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504"} Oct 31 12:52:29.016065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.015940 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" event={"ID":"7e2ba614-7a90-4d12-9920-10583af4294a","Type":"ContainerDied","Data":"1423cd44bc2385c0bb85ccf72215c67f096e56a25c5822be90ccba0187ecadac"} Oct 31 12:52:29.016065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.015962 5074 scope.go:122] "RemoveContainer" containerID="fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504" Oct 31 12:52:29.016065 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.016062 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk" Oct 31 12:52:29.018774 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.018747 5074 generic.go:356] "Generic (PLEG): container finished" podID="2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" containerID="fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666" exitCode=0 Oct 31 12:52:29.018908 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.018888 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" event={"ID":"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e","Type":"ContainerDied","Data":"fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666"} Oct 31 12:52:29.018977 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.018935 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" event={"ID":"2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e","Type":"ContainerDied","Data":"8d0e16a10abd689004face68913657f5182a9c56b309448e11c9c4583f191b13"} Oct 31 12:52:29.019060 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.019046 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk" Oct 31 12:52:29.031122 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.030877 5074 scope.go:122] "RemoveContainer" containerID="fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504" Oct 31 12:52:29.032116 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:29.032091 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504\": container with ID starting with fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504 not found: ID does not exist" containerID="fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504" Oct 31 12:52:29.032220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.032127 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504"} err="failed to get container status \"fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504\": rpc error: code = NotFound desc = could not find container \"fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504\": container with ID starting with fc75ee443a85adfe0d1a18ec642a50f29e881a3f9f8cf9539fc1445e3cb0a504 not found: ID does not exist" Oct 31 12:52:29.032220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.032147 5074 scope.go:122] "RemoveContainer" containerID="fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666" Oct 31 12:52:29.041533 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.041516 5074 scope.go:122] "RemoveContainer" containerID="fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666" Oct 31 12:52:29.041835 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:29.041815 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666\": container with ID starting with fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666 not found: ID does not exist" containerID="fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666" Oct 31 12:52:29.041902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.041841 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666"} err="failed to get container status \"fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666\": rpc error: code = NotFound desc = could not find container \"fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666\": container with ID starting with fc1c6c3f02049be1445e06a0b0733df5bbf647d3b63d58b441e499c2a982e666 not found: ID does not exist" Oct 31 12:52:29.045962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.045937 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk"] Oct 31 12:52:29.047820 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.047801 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1abc54b4-b44b944cf-w2bxk"] Oct 31 12:52:29.059792 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.059765 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk"] Oct 31 12:52:29.065680 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.065656 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1abc54b4-69b78587b7-wd5lk"] Oct 31 12:52:29.290658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.289617 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:29.447917 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.447877 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj"] Oct 31 12:52:29.448497 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448480 5074 cpu_manager.go:414] "containerMap: removing container" podUID="22dcca93-c9ba-4ba8-87c1-17cab7e044dd" containerName="revoke-keys" Oct 31 12:52:29.448497 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448498 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dcca93-c9ba-4ba8-87c1-17cab7e044dd" containerName="revoke-keys" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448527 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ba614-7a90-4d12-9920-10583af4294a" containerName="payload-processing" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448535 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ba614-7a90-4d12-9920-10583af4294a" containerName="payload-processing" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448546 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" containerName="payload-pre-processing" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448555 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" containerName="payload-pre-processing" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448567 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0377fb34-98f0-4e22-9f4e-29287c7970a1" containerName="payload-pre-processing" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448574 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0377fb34-98f0-4e22-9f4e-29287c7970a1" containerName="payload-pre-processing" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448598 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2e3cc385-d3ae-43e2-aa72-fffec8dc725f" containerName="revoke-keys" Oct 31 12:52:29.448690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448605 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3cc385-d3ae-43e2-aa72-fffec8dc725f" containerName="revoke-keys" Oct 31 12:52:29.449091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448703 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ba614-7a90-4d12-9920-10583af4294a" containerName="payload-processing" Oct 31 12:52:29.449091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448714 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" containerName="payload-pre-processing" Oct 31 12:52:29.449091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448724 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0377fb34-98f0-4e22-9f4e-29287c7970a1" containerName="payload-pre-processing" Oct 31 12:52:29.449091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448736 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3cc385-d3ae-43e2-aa72-fffec8dc725f" containerName="revoke-keys" Oct 31 12:52:29.449091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.448748 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dcca93-c9ba-4ba8-87c1-17cab7e044dd" containerName="revoke-keys" Oct 31 12:52:29.454234 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.453810 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.465708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.465654 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-107955-openshift-default-dockercfg-qq5mx\"" Oct 31 12:52:29.468506 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.468484 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj"] Oct 31 12:52:29.546641 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.546549 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.546641 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.546601 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.546815 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.546772 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.546851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.546826 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sln2l\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-kube-api-access-sln2l\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.546925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.546907 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.546974 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.546948 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.547038 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.547023 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.547120 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.547063 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.547120 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.547096 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648591 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648547 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648609 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648700 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648739 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648772 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648809 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648842 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.648919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648877 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.649007 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.648923 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sln2l\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-kube-api-access-sln2l\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.650109 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.650085 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.651041 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.651016 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.651475 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.651449 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.651625 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.651494 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.651758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.651734 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.651962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.651938 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.654182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.654159 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.662707 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.662686 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.662914 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.662895 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sln2l\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-kube-api-access-sln2l\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.823420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.823328 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:29.943580 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.943545 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e" path="/var/lib/kubelet/pods/2c33f2ac-7d6c-4e7c-bff3-1c6df9c9d83e/volumes" Oct 31 12:52:29.944149 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:29.944128 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2ba614-7a90-4d12-9920-10583af4294a" path="/var/lib/kubelet/pods/7e2ba614-7a90-4d12-9920-10583af4294a/volumes" Oct 31 12:52:30.191711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:30.180340 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv"] Oct 31 12:52:30.212962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:30.210332 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj"] Oct 31 12:52:30.222008 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:30.212930 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39ca4bf_f2b7_4615_93d1_ace6e650716d.slice/crio-c0dba4a64e0f8e3b96a95f96360a6d572298297b9ea4221325cd4cc56af14fb4 WatchSource:0}: Error finding container c0dba4a64e0f8e3b96a95f96360a6d572298297b9ea4221325cd4cc56af14fb4: Status 404 returned error can't find the container with id c0dba4a64e0f8e3b96a95f96360a6d572298297b9ea4221325cd4cc56af14fb4 Oct 31 12:52:30.222008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:30.214936 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:30.222008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:30.215320 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:30.222008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:30.215373 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:30.238772 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:30.238734 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp"] Oct 31 12:52:31.034961 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.034922 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" event={"ID":"c39ca4bf-f2b7-4615-93d1-ace6e650716d","Type":"ContainerStarted","Data":"af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b"} Oct 31 12:52:31.035422 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.034992 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" event={"ID":"c39ca4bf-f2b7-4615-93d1-ace6e650716d","Type":"ContainerStarted","Data":"c0dba4a64e0f8e3b96a95f96360a6d572298297b9ea4221325cd4cc56af14fb4"} Oct 31 12:52:31.110744 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.110688 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" podStartSLOduration=2.110671973 podStartE2EDuration="2.110671973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:31.066958104 +0000 UTC m=+1245.244676087" watchObservedRunningTime="2026-10-31 12:52:31.110671973 +0000 UTC m=+1245.288389956" Oct 31 12:52:31.113005 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.112975 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6"] Oct 31 12:52:31.114824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.114800 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d48f454a-dh2b6"] Oct 31 12:52:31.824369 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.824332 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:31.831256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.831226 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:31.941423 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:31.941381 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dcca93-c9ba-4ba8-87c1-17cab7e044dd" path="/var/lib/kubelet/pods/22dcca93-c9ba-4ba8-87c1-17cab7e044dd/volumes" Oct 31 12:52:32.039946 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.039914 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:32.041044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.041019 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:32.263616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.263560 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" podUID="d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" containerName="payload-processing" containerID="cri-o://5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421" gracePeriod=25 Oct 31 12:52:32.393959 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.393167 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b"] Oct 31 12:52:32.396672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.396618 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-785c04b9-f877b"] Oct 31 12:52:32.597061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.597033 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:52:32.681568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.681526 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-plugins-config-volume\") pod \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " Oct 31 12:52:32.681802 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.681698 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-kube-api-access-8g86j\" (UniqueName: \"kubernetes.io/projected/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-kube-api-access-8g86j\") pod \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\" (UID: \"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2\") " Oct 31 12:52:32.682345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.682279 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-plugins-config-volume" pod "d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" (UID: "d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:32.684732 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.684701 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-kube-api-access-8g86j" pod "d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" (UID: "d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2"). InnerVolumeSpecName "kube-api-access-8g86j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:32.782834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.782788 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g86j\" (UniqueName: \"kubernetes.io/projected/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-kube-api-access-8g86j\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:32.782834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:32.782821 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:33.047404 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.047362 5074 generic.go:356] "Generic (PLEG): container finished" podID="d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" containerID="5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421" exitCode=0 Oct 31 12:52:33.048099 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.047410 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" Oct 31 12:52:33.048099 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.047441 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" event={"ID":"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2","Type":"ContainerDied","Data":"5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421"} Oct 31 12:52:33.048099 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.047499 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44" event={"ID":"d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2","Type":"ContainerDied","Data":"a2663f1c9e96342c9b18b1c96f8fde3e272568e4d75362d088b92ac5da3660c0"} Oct 31 12:52:33.048099 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.047527 5074 scope.go:122] "RemoveContainer" containerID="5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421" Oct 31 12:52:33.057712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.057691 5074 scope.go:122] "RemoveContainer" containerID="5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421" Oct 31 12:52:33.058196 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:33.058170 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421\": container with ID starting with 5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421 not found: ID does not exist" containerID="5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421" Oct 31 12:52:33.058397 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.058371 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421"} err="failed to get container status \"5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421\": rpc error: code = NotFound desc = could not find container \"5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421\": container with ID starting with 5fd811988a2032f71e27435ac2d29bd24d75d881a4ca5372627b20d8dd4dc421 not found: ID does not exist" Oct 31 12:52:33.075353 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.075330 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44"] Oct 31 12:52:33.080468 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.080445 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d48f454a-5fff859755-nmg44"] Oct 31 12:52:33.942506 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.942463 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3cc385-d3ae-43e2-aa72-fffec8dc725f" path="/var/lib/kubelet/pods/2e3cc385-d3ae-43e2-aa72-fffec8dc725f/volumes" Oct 31 12:52:33.942976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:33.942953 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" path="/var/lib/kubelet/pods/d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2/volumes" Oct 31 12:52:34.311432 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:34.311391 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:34.941023 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:34.940988 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7"] Oct 31 12:52:34.941427 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:34.941414 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" containerName="payload-processing" Oct 31 12:52:34.941474 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:34.941427 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" containerName="payload-processing" Oct 31 12:52:34.941507 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:34.941490 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d411f3a0-f6f9-41df-a6f2-8a6e6df4c1c2" containerName="payload-processing" Oct 31 12:52:34.947395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:34.947373 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:34.961610 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:34.961192 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7"] Oct 31 12:52:35.029655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.029452 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1cc28c0-cb1c-42f0-b194-b370e52f8490-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:35.029655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.029575 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlrf\" (UniqueName: \"kubernetes.io/projected/d1cc28c0-cb1c-42f0-b194-b370e52f8490-kube-api-access-pvlrf\") pod \"payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:35.046625 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.046591 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-107955\"" Oct 31 12:52:35.047108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.047089 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-107955-dockercfg-dp8wc\"" Oct 31 12:52:35.062754 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.062723 5074 generic.go:356] "Generic (PLEG): container finished" podID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerID="aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f" exitCode=0 Oct 31 12:52:35.062910 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.062792 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" event={"ID":"230e0627-e62e-4f58-ac8e-91dc89cf2319","Type":"ContainerDied","Data":"aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f"} Oct 31 12:52:35.063190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.063175 5074 scope.go:122] "RemoveContainer" containerID="aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f" Oct 31 12:52:35.064458 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.064436 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:35.064512 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.064497 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:35.064547 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.064531 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:35.134296 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.134272 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1cc28c0-cb1c-42f0-b194-b370e52f8490-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:35.134410 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.134346 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlrf\" (UniqueName: \"kubernetes.io/projected/d1cc28c0-cb1c-42f0-b194-b370e52f8490-kube-api-access-pvlrf\") pod \"payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:35.135070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.135048 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1cc28c0-cb1c-42f0-b194-b370e52f8490-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:35.147714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.147690 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlrf\" (UniqueName: \"kubernetes.io/projected/d1cc28c0-cb1c-42f0-b194-b370e52f8490-kube-api-access-pvlrf\") pod \"payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:35.180705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.180657 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" podUID="30279950-6bdb-41ed-b712-c2fa9e2473b6" containerName="payload-processing" containerID="cri-o://fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7" gracePeriod=25 Oct 31 12:52:35.239573 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.239528 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" podUID="bc39df69-69dd-4993-a31d-47938f9e5cf5" containerName="payload-pre-processing" containerID="cri-o://d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e" gracePeriod=25 Oct 31 12:52:35.265393 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.265361 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:35.546918 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.546878 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:52:35.614856 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:35.614819 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cc28c0_cb1c_42f0_b194_b370e52f8490.slice/crio-56d301f3287bef92dea6394e7cdde01bdd8c509d2ecd71492f2c4377c4803480 WatchSource:0}: Error finding container 56d301f3287bef92dea6394e7cdde01bdd8c509d2ecd71492f2c4377c4803480: Status 404 returned error can't find the container with id 56d301f3287bef92dea6394e7cdde01bdd8c509d2ecd71492f2c4377c4803480 Oct 31 12:52:35.648310 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.642667 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7"] Oct 31 12:52:35.655161 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.654330 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30279950-6bdb-41ed-b712-c2fa9e2473b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30279950-6bdb-41ed-b712-c2fa9e2473b6-plugins-config-volume\") pod \"30279950-6bdb-41ed-b712-c2fa9e2473b6\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " Oct 31 12:52:35.655161 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.654433 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30279950-6bdb-41ed-b712-c2fa9e2473b6-kube-api-access-8mhrv\" (UniqueName: \"kubernetes.io/projected/30279950-6bdb-41ed-b712-c2fa9e2473b6-kube-api-access-8mhrv\") pod \"30279950-6bdb-41ed-b712-c2fa9e2473b6\" (UID: \"30279950-6bdb-41ed-b712-c2fa9e2473b6\") " Oct 31 12:52:35.655962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.655932 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30279950-6bdb-41ed-b712-c2fa9e2473b6-plugins-config-volume" pod "30279950-6bdb-41ed-b712-c2fa9e2473b6" (UID: "30279950-6bdb-41ed-b712-c2fa9e2473b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:35.657877 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.657856 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30279950-6bdb-41ed-b712-c2fa9e2473b6-kube-api-access-8mhrv" pod "30279950-6bdb-41ed-b712-c2fa9e2473b6" (UID: "30279950-6bdb-41ed-b712-c2fa9e2473b6"). InnerVolumeSpecName "kube-api-access-8mhrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:35.665069 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.664805 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:52:35.755852 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.755814 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc39df69-69dd-4993-a31d-47938f9e5cf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39df69-69dd-4993-a31d-47938f9e5cf5-plugins-config-volume\") pod \"bc39df69-69dd-4993-a31d-47938f9e5cf5\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " Oct 31 12:52:35.756401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.755904 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc39df69-69dd-4993-a31d-47938f9e5cf5-kube-api-access-wtmv6\" (UniqueName: \"kubernetes.io/projected/bc39df69-69dd-4993-a31d-47938f9e5cf5-kube-api-access-wtmv6\") pod \"bc39df69-69dd-4993-a31d-47938f9e5cf5\" (UID: \"bc39df69-69dd-4993-a31d-47938f9e5cf5\") " Oct 31 12:52:35.756401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.756294 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30279950-6bdb-41ed-b712-c2fa9e2473b6-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:35.756401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.756321 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mhrv\" (UniqueName: \"kubernetes.io/projected/30279950-6bdb-41ed-b712-c2fa9e2473b6-kube-api-access-8mhrv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:35.757250 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.757215 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc39df69-69dd-4993-a31d-47938f9e5cf5-plugins-config-volume" pod "bc39df69-69dd-4993-a31d-47938f9e5cf5" (UID: "bc39df69-69dd-4993-a31d-47938f9e5cf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:52:35.759015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.758986 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc39df69-69dd-4993-a31d-47938f9e5cf5-kube-api-access-wtmv6" pod "bc39df69-69dd-4993-a31d-47938f9e5cf5" (UID: "bc39df69-69dd-4993-a31d-47938f9e5cf5"). InnerVolumeSpecName "kube-api-access-wtmv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:52:35.857212 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.857173 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc39df69-69dd-4993-a31d-47938f9e5cf5-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:35.857212 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:35.857215 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtmv6\" (UniqueName: \"kubernetes.io/projected/bc39df69-69dd-4993-a31d-47938f9e5cf5-kube-api-access-wtmv6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:52:36.078136 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.078012 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" event={"ID":"230e0627-e62e-4f58-ac8e-91dc89cf2319","Type":"ContainerStarted","Data":"5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06"} Oct 31 12:52:36.086902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.086850 5074 generic.go:356] "Generic (PLEG): container finished" podID="bc39df69-69dd-4993-a31d-47938f9e5cf5" containerID="d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e" exitCode=0 Oct 31 12:52:36.087089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.087004 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" event={"ID":"bc39df69-69dd-4993-a31d-47938f9e5cf5","Type":"ContainerDied","Data":"d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e"} Oct 31 12:52:36.087089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.087054 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" event={"ID":"bc39df69-69dd-4993-a31d-47938f9e5cf5","Type":"ContainerDied","Data":"8957a5b09a7e1bd767e8ce5b84b408538058133a36cf08a1d76ec19425ea8144"} Oct 31 12:52:36.087089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.087085 5074 scope.go:122] "RemoveContainer" containerID="d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e" Oct 31 12:52:36.087249 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.087220 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp" Oct 31 12:52:36.094078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.090426 5074 generic.go:356] "Generic (PLEG): container finished" podID="30279950-6bdb-41ed-b712-c2fa9e2473b6" containerID="fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7" exitCode=0 Oct 31 12:52:36.094078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.090513 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" event={"ID":"30279950-6bdb-41ed-b712-c2fa9e2473b6","Type":"ContainerDied","Data":"fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7"} Oct 31 12:52:36.094078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.090554 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" event={"ID":"30279950-6bdb-41ed-b712-c2fa9e2473b6","Type":"ContainerDied","Data":"f421c9db8b8c6798d4a28b2ddb97ed3ecde7cf530d63f8b92ca4017136221fde"} Oct 31 12:52:36.094078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.090648 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv" Oct 31 12:52:36.094078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.092739 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" event={"ID":"d1cc28c0-cb1c-42f0-b194-b370e52f8490","Type":"ContainerStarted","Data":"ebd07b745bad1da57fa73d41873bb8cb1e6749bff836f0add78431c3ef7e2b85"} Oct 31 12:52:36.094078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.092785 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" event={"ID":"d1cc28c0-cb1c-42f0-b194-b370e52f8490","Type":"ContainerStarted","Data":"56d301f3287bef92dea6394e7cdde01bdd8c509d2ecd71492f2c4377c4803480"} Oct 31 12:52:36.094078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.093497 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:36.116589 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.116566 5074 scope.go:122] "RemoveContainer" containerID="d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e" Oct 31 12:52:36.116915 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:36.116890 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e\": container with ID starting with d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e not found: ID does not exist" containerID="d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e" Oct 31 12:52:36.116973 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.116924 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e"} err="failed to get container status \"d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e\": rpc error: code = NotFound desc = could not find container \"d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e\": container with ID starting with d0f98c3f147dae7222a3c625b3ef02fa70aa43dfffd165943921104497f0526e not found: ID does not exist" Oct 31 12:52:36.116973 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.116948 5074 scope.go:122] "RemoveContainer" containerID="fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7" Oct 31 12:52:36.127611 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.127590 5074 scope.go:122] "RemoveContainer" containerID="fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7" Oct 31 12:52:36.128135 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:36.128111 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7\": container with ID starting with fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7 not found: ID does not exist" containerID="fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7" Oct 31 12:52:36.128233 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.128141 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7"} err="failed to get container status \"fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7\": rpc error: code = NotFound desc = could not find container \"fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7\": container with ID starting with fc8c6bec0537a3a45313524789ae4c8305774c69c9268a848ec964926a4032c7 not found: ID does not exist" Oct 31 12:52:36.136142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.136088 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" podStartSLOduration=2.136072177 podStartE2EDuration="2.136072177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:36.133570931 +0000 UTC m=+1250.311288916" watchObservedRunningTime="2026-10-31 12:52:36.136072177 +0000 UTC m=+1250.313790162" Oct 31 12:52:36.151820 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.151793 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp"] Oct 31 12:52:36.160189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.160168 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-785c04b9-cfcfc467-nslnp"] Oct 31 12:52:36.205756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.205687 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv"] Oct 31 12:52:36.208840 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.208806 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-785c04b9-6557b48c55-xkmtv"] Oct 31 12:52:36.726727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.726691 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:36.727139 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.726829 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:52:36.727139 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:36.726885 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 12:52:37.349608 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.349562 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4"] Oct 31 12:52:37.350092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.350070 5074 cpu_manager.go:414] "containerMap: removing container" podUID="bc39df69-69dd-4993-a31d-47938f9e5cf5" containerName="payload-pre-processing" Oct 31 12:52:37.350193 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.350093 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc39df69-69dd-4993-a31d-47938f9e5cf5" containerName="payload-pre-processing" Oct 31 12:52:37.350193 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.350157 5074 cpu_manager.go:414] "containerMap: removing container" podUID="30279950-6bdb-41ed-b712-c2fa9e2473b6" containerName="payload-processing" Oct 31 12:52:37.350193 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.350166 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30279950-6bdb-41ed-b712-c2fa9e2473b6" containerName="payload-processing" Oct 31 12:52:37.350336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.350291 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc39df69-69dd-4993-a31d-47938f9e5cf5" containerName="payload-pre-processing" Oct 31 12:52:37.350336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.350313 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="30279950-6bdb-41ed-b712-c2fa9e2473b6" containerName="payload-processing" Oct 31 12:52:37.354944 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.354923 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.361874 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.361835 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-3dd1e5-dockercfg-fj5hr\"" Oct 31 12:52:37.362426 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.362384 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-3dd1e5\"" Oct 31 12:52:37.364813 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.364786 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4"] Oct 31 12:52:37.400220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.400182 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb"] Oct 31 12:52:37.404070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.404045 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.415020 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.414995 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb"] Oct 31 12:52:37.516147 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.516114 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccc2m\" (UniqueName: \"kubernetes.io/projected/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-kube-api-access-ccc2m\") pod \"payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.516428 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.516406 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.516523 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.516462 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvb8\" (UniqueName: \"kubernetes.io/projected/d42989ca-a6d6-4900-94ee-269a7c92ed94-kube-api-access-9jvb8\") pod \"payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.516607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.516578 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42989ca-a6d6-4900-94ee-269a7c92ed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.617269 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.617168 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.617269 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.617222 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvb8\" (UniqueName: \"kubernetes.io/projected/d42989ca-a6d6-4900-94ee-269a7c92ed94-kube-api-access-9jvb8\") pod \"payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.617269 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.617263 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42989ca-a6d6-4900-94ee-269a7c92ed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.617566 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.617294 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccc2m\" (UniqueName: \"kubernetes.io/projected/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-kube-api-access-ccc2m\") pod \"payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.618079 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.618053 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.618186 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.618164 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42989ca-a6d6-4900-94ee-269a7c92ed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.633222 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.633189 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccc2m\" (UniqueName: \"kubernetes.io/projected/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-kube-api-access-ccc2m\") pod \"payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.634594 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.634573 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvb8\" (UniqueName: \"kubernetes.io/projected/d42989ca-a6d6-4900-94ee-269a7c92ed94-kube-api-access-9jvb8\") pod \"payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.688657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.685811 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:37.718116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.718083 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:37.726697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.726666 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:52:37.726863 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.726723 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 12:52:37.938223 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.938144 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30279950-6bdb-41ed-b712-c2fa9e2473b6" path="/var/lib/kubelet/pods/30279950-6bdb-41ed-b712-c2fa9e2473b6/volumes" Oct 31 12:52:37.938692 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.938670 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc39df69-69dd-4993-a31d-47938f9e5cf5" path="/var/lib/kubelet/pods/bc39df69-69dd-4993-a31d-47938f9e5cf5/volumes" Oct 31 12:52:37.996656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:37.996544 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4"] Oct 31 12:52:38.000174 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:38.000136 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42989ca_a6d6_4900_94ee_269a7c92ed94.slice/crio-ff2cbe8a1fb618a9261da19ee283e868266742451afdda08f491ff2b43542048 WatchSource:0}: Error finding container ff2cbe8a1fb618a9261da19ee283e868266742451afdda08f491ff2b43542048: Status 404 returned error can't find the container with id ff2cbe8a1fb618a9261da19ee283e868266742451afdda08f491ff2b43542048 Oct 31 12:52:38.104463 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.104068 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb"] Oct 31 12:52:38.106163 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:38.106107 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ad5b2c_ee99_4d37_8ff0_4025ee6426b7.slice/crio-eb073e40d8dc5c9fbfb35288842bf25bf6499a0bdf84972a15391b3dd750e720 WatchSource:0}: Error finding container eb073e40d8dc5c9fbfb35288842bf25bf6499a0bdf84972a15391b3dd750e720: Status 404 returned error can't find the container with id eb073e40d8dc5c9fbfb35288842bf25bf6499a0bdf84972a15391b3dd750e720 Oct 31 12:52:38.125784 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.124894 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" event={"ID":"d42989ca-a6d6-4900-94ee-269a7c92ed94","Type":"ContainerStarted","Data":"0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d"} Oct 31 12:52:38.125784 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.124958 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" event={"ID":"d42989ca-a6d6-4900-94ee-269a7c92ed94","Type":"ContainerStarted","Data":"ff2cbe8a1fb618a9261da19ee283e868266742451afdda08f491ff2b43542048"} Oct 31 12:52:38.125784 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.125739 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:38.127248 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.127191 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" event={"ID":"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7","Type":"ContainerStarted","Data":"eb073e40d8dc5c9fbfb35288842bf25bf6499a0bdf84972a15391b3dd750e720"} Oct 31 12:52:38.149562 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.149518 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" podStartSLOduration=1.149505308 podStartE2EDuration="1.149505308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:38.14846134 +0000 UTC m=+1252.326179324" watchObservedRunningTime="2026-10-31 12:52:38.149505308 +0000 UTC m=+1252.327223291" Oct 31 12:52:38.727751 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.727706 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:52:38.728185 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:38.727777 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 12:52:39.139161 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:39.139066 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" event={"ID":"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7","Type":"ContainerStarted","Data":"459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d"} Oct 31 12:52:39.139498 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:39.139478 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:39.165670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:39.165590 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" podStartSLOduration=2.165561083 podStartE2EDuration="2.165561083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:39.163940142 +0000 UTC m=+1253.341658120" watchObservedRunningTime="2026-10-31 12:52:39.165561083 +0000 UTC m=+1253.343279073" Oct 31 12:52:39.727030 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:39.726996 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:52:39.727183 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:39.727048 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 12:52:40.726247 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:40.726211 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 12:52:40.726744 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:40.726264 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 12:52:41.269492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.269453 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc"] Oct 31 12:52:41.273509 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.273486 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.287985 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.287945 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"simulator-w3-47bcd9-kserve-self-signed-certs\"" Oct 31 12:52:41.288128 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.288092 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"default-dockercfg-pcs5t\"" Oct 31 12:52:41.291207 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.291178 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc"] Oct 31 12:52:41.340866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.340834 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"kube-root-ca.crt\"" Oct 31 12:52:41.341055 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.340966 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"openshift-service-ca.crt\"" Oct 31 12:52:41.470088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.470034 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94beb7c-137f-4c8c-b08e-cf95d981f725-tls-certs\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.470283 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.470167 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-model-cache\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.470283 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.470209 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-tmp-dir\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.470283 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.470245 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-dshm\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.470283 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.470272 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xbh\" (UniqueName: \"kubernetes.io/projected/d94beb7c-137f-4c8c-b08e-cf95d981f725-kube-api-access-p6xbh\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.470537 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.470297 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-home\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.572672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.571828 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-model-cache\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.572672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.571885 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-tmp-dir\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.572672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.571921 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-dshm\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.572672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.571950 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xbh\" (UniqueName: \"kubernetes.io/projected/d94beb7c-137f-4c8c-b08e-cf95d981f725-kube-api-access-p6xbh\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.572672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.571978 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-home\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.572672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.572051 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94beb7c-137f-4c8c-b08e-cf95d981f725-tls-certs\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.572672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.572455 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-model-cache\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.573168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.573146 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-tmp-dir\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.573591 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.573572 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-home\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.578667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.578625 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94beb7c-137f-4c8c-b08e-cf95d981f725-tls-certs\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.585885 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.585854 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-dshm\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.586019 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.585908 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xbh\" (UniqueName: \"kubernetes.io/projected/d94beb7c-137f-4c8c-b08e-cf95d981f725-kube-api-access-p6xbh\") pod \"simulator-w3-47bcd9-kserve-79f9d74765-5gtgc\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.615495 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.615446 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:41.733673 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.733517 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:41.734159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.734079 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:41.734916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.734894 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 12:52:41.899556 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:41.899458 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc"] Oct 31 12:52:41.903420 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:41.903391 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd94beb7c_137f_4c8c_b08e_cf95d981f725.slice/crio-96ee085234ade5aedc816b94f55e4998ddfddb7025c6b05992d4b4a25c81dfa1 WatchSource:0}: Error finding container 96ee085234ade5aedc816b94f55e4998ddfddb7025c6b05992d4b4a25c81dfa1: Status 404 returned error can't find the container with id 96ee085234ade5aedc816b94f55e4998ddfddb7025c6b05992d4b4a25c81dfa1 Oct 31 12:52:42.158068 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:42.157988 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" event={"ID":"d94beb7c-137f-4c8c-b08e-cf95d981f725","Type":"ContainerStarted","Data":"96ee085234ade5aedc816b94f55e4998ddfddb7025c6b05992d4b4a25c81dfa1"} Oct 31 12:52:43.834915 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:43.834870 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:44.174681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:44.174580 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" event={"ID":"d94beb7c-137f-4c8c-b08e-cf95d981f725","Type":"ContainerStarted","Data":"8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6"} Oct 31 12:52:44.204420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:44.204369 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podStartSLOduration=1.784292502 podStartE2EDuration="3.204354862s" totalImagesPullingTime="1.42006236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:44.202667021 +0000 UTC m=+1258.380385007" watchObservedRunningTime="2026-10-31 12:52:44.204354862 +0000 UTC m=+1258.382072846" Oct 31 12:52:44.615985 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:44.615914 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:48.130750 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:48.130717 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 12:52:48.766652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:48.766595 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:50.146841 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:50.146814 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 12:52:50.147316 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:50.147196 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 12:52:51.616685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:51.616645 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:51.630113 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:51.630083 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:52.310169 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:52.310134 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:52:55.749286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.749251 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x"] Oct 31 12:52:55.753393 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.753370 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.795612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.795575 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"default-dockercfg-gxcbm\"" Oct 31 12:52:55.795612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.795608 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"kube-root-ca.crt\"" Oct 31 12:52:55.796412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.796389 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"simulator-w0-107955-kserve-self-signed-certs\"" Oct 31 12:52:55.803321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.803294 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x"] Oct 31 12:52:55.813828 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.813794 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"openshift-service-ca.crt\"" Oct 31 12:52:55.843199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.843165 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzhh\" (UniqueName: \"kubernetes.io/projected/be21f7d2-5f5f-4592-afdd-591be409c854-kube-api-access-6qzhh\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.843526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.843493 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21f7d2-5f5f-4592-afdd-591be409c854-tls-certs\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.843735 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.843713 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-tmp-dir\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.843951 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.843927 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-home\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.844072 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.843980 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-model-cache\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.844072 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.844010 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-dshm\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.944994 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.944952 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzhh\" (UniqueName: \"kubernetes.io/projected/be21f7d2-5f5f-4592-afdd-591be409c854-kube-api-access-6qzhh\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945183 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945010 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21f7d2-5f5f-4592-afdd-591be409c854-tls-certs\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945183 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945076 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-tmp-dir\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945183 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945159 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-home\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945361 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945208 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-model-cache\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945361 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945237 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-dshm\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945611 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945576 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-tmp-dir\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945611 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945599 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-model-cache\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.945811 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.945654 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-home\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.947490 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.947466 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-dshm\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.947680 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.947661 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21f7d2-5f5f-4592-afdd-591be409c854-tls-certs\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:55.984232 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:52:55.984195 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:52:55.986731 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:55.986709 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzhh\" (UniqueName: \"kubernetes.io/projected/be21f7d2-5f5f-4592-afdd-591be409c854-kube-api-access-6qzhh\") pod \"simulator-w0-107955-kserve-674f8fcd98-25x5x\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:56.067831 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.066610 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:56.289794 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.289764 5074 generic.go:356] "Generic (PLEG): container finished" podID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerID="af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b" exitCode=0 Oct 31 12:52:56.289996 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.289835 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" event={"ID":"c39ca4bf-f2b7-4615-93d1-ace6e650716d","Type":"ContainerDied","Data":"af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b"} Oct 31 12:52:56.290249 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.290232 5074 scope.go:122] "RemoveContainer" containerID="af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b" Oct 31 12:52:56.291766 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.291737 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:56.291889 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.291821 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:56.291889 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.291872 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:52:56.368739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:56.368708 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x"] Oct 31 12:52:56.372616 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:52:56.372588 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe21f7d2_5f5f_4592_afdd_591be409c854.slice/crio-3c7817c1995b4a935c3c9931b89bf859b0553b147daa117e88789f26a1e23d4f WatchSource:0}: Error finding container 3c7817c1995b4a935c3c9931b89bf859b0553b147daa117e88789f26a1e23d4f: Status 404 returned error can't find the container with id 3c7817c1995b4a935c3c9931b89bf859b0553b147daa117e88789f26a1e23d4f Oct 31 12:52:57.307916 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:57.307880 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" event={"ID":"c39ca4bf-f2b7-4615-93d1-ace6e650716d","Type":"ContainerStarted","Data":"e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579"} Oct 31 12:52:57.309996 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:57.309970 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" event={"ID":"be21f7d2-5f5f-4592-afdd-591be409c854","Type":"ContainerStarted","Data":"97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187"} Oct 31 12:52:57.310157 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:57.310003 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" event={"ID":"be21f7d2-5f5f-4592-afdd-591be409c854","Type":"ContainerStarted","Data":"3c7817c1995b4a935c3c9931b89bf859b0553b147daa117e88789f26a1e23d4f"} Oct 31 12:52:57.361755 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:57.361703 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podStartSLOduration=2.361690448 podStartE2EDuration="2.361690448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:57.361087418 +0000 UTC m=+1271.538805404" watchObservedRunningTime="2026-10-31 12:52:57.361690448 +0000 UTC m=+1271.539408431" Oct 31 12:52:57.824927 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:57.824887 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:52:57.826369 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:57.826331 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 12:52:57.826497 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:57.826398 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 12:52:58.068362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:58.068333 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:52:58.824903 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:58.824863 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 12:52:58.825310 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:58.824946 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 12:52:58.914655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:58.913204 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:52:59.824704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:59.824668 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 12:52:59.824889 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:52:59.824734 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 12:53:00.824111 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:00.824076 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 12:53:00.824521 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:00.824128 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 12:53:01.825126 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:01.825085 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 12:53:01.825664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:01.825149 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 12:53:02.040776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:02.040743 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:53:02.982218 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:02.982190 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:53:02.984293 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:02.984263 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 12:53:03.739155 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.739122 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x"] Oct 31 12:53:03.742968 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.742943 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.752842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.752814 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"default-dockercfg-hzxqn\"" Oct 31 12:53:03.753083 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.753061 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"openshift-service-ca.crt\"" Oct 31 12:53:03.753211 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.753158 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"simulator-w2-3dd1e5-kserve-self-signed-certs\"" Oct 31 12:53:03.753278 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.753227 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"kube-root-ca.crt\"" Oct 31 12:53:03.754379 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.754356 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x"] Oct 31 12:53:03.860230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.860188 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-home\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.860421 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.860306 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-model-cache\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.860421 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.860360 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tls-certs\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.860421 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.860417 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tmp-dir\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.860604 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.860501 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d89d\" (UniqueName: \"kubernetes.io/projected/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-kube-api-access-7d89d\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.860604 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.860532 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-dshm\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961290 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961260 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d89d\" (UniqueName: \"kubernetes.io/projected/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-kube-api-access-7d89d\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961297 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-dshm\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961334 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-home\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961375 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-model-cache\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961403 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tls-certs\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961449 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tmp-dir\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961794 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961778 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tmp-dir\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.961843 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.961811 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-home\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.962210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.962184 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-model-cache\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.963951 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.963928 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-dshm\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.964203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.964182 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tls-certs\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:03.976344 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:03.976317 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d89d\" (UniqueName: \"kubernetes.io/projected/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-kube-api-access-7d89d\") pod \"simulator-w2-3dd1e5-kserve-6569f94675-2884x\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:04.057205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:04.057118 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:04.337824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:04.337801 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x"] Oct 31 12:53:04.339856 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:53:04.339826 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85a5c9b7_eada_4d26_8e3a_e07bb9215c53.slice/crio-e8e490a34534e36f3e2f11b36d936ede11af20c77de5e96e0aac0672f479085a WatchSource:0}: Error finding container e8e490a34534e36f3e2f11b36d936ede11af20c77de5e96e0aac0672f479085a: Status 404 returned error can't find the container with id e8e490a34534e36f3e2f11b36d936ede11af20c77de5e96e0aac0672f479085a Oct 31 12:53:04.352927 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:04.352899 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" event={"ID":"85a5c9b7-eada-4d26-8e3a-e07bb9215c53","Type":"ContainerStarted","Data":"e8e490a34534e36f3e2f11b36d936ede11af20c77de5e96e0aac0672f479085a"} Oct 31 12:53:05.359581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:05.359542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" event={"ID":"85a5c9b7-eada-4d26-8e3a-e07bb9215c53","Type":"ContainerStarted","Data":"3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a"} Oct 31 12:53:05.380721 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:05.380670 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podStartSLOduration=2.38065691 podStartE2EDuration="2.38065691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:05.378814085 +0000 UTC m=+1279.556532069" watchObservedRunningTime="2026-10-31 12:53:05.38065691 +0000 UTC m=+1279.558374890" Oct 31 12:53:06.058308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:06.058260 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:06.068268 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:06.068207 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:53:06.090532 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:06.090503 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:53:06.378949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:06.378855 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:53:06.389593 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:06.389562 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:53:10.503071 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.503036 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv"] Oct 31 12:53:10.508756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.508726 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.517052 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.517013 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv"] Oct 31 12:53:10.522378 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.522356 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"premium-w3-47bcd9-kserve-self-signed-certs\"" Oct 31 12:53:10.536902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.536864 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-home\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.536902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.536900 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-model-cache\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.537127 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.536925 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-dshm\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.537127 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.537080 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tls-certs\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.537235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.537132 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4rz\" (UniqueName: \"kubernetes.io/projected/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-kube-api-access-5b4rz\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.537235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.537207 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tmp-dir\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.638838 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-home\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.638887 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-model-cache\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.638921 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-dshm\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.638987 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tls-certs\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.639024 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4rz\" (UniqueName: \"kubernetes.io/projected/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-kube-api-access-5b4rz\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.639102 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tmp-dir\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.639607 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tmp-dir\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.639890 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-home\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.640517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.640183 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-model-cache\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.665872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.665829 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-dshm\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.682707 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.666569 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tls-certs\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.682707 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.678375 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4rz\" (UniqueName: \"kubernetes.io/projected/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-kube-api-access-5b4rz\") pod \"premium-w3-47bcd9-kserve-585b654656-7v5xv\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:10.826001 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:10.825902 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:11.108936 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:11.108905 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv"] Oct 31 12:53:11.110547 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:53:11.110515 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a85d43_3cca_48ff_b8a3_f8de2beff5b1.slice/crio-fccc3fb4d61266d487dbd43a12a6c916032f4333630fef0261aa99eaefbe4f84 WatchSource:0}: Error finding container fccc3fb4d61266d487dbd43a12a6c916032f4333630fef0261aa99eaefbe4f84: Status 404 returned error can't find the container with id fccc3fb4d61266d487dbd43a12a6c916032f4333630fef0261aa99eaefbe4f84 Oct 31 12:53:11.402851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:11.402761 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" event={"ID":"90a85d43-3cca-48ff-b8a3-f8de2beff5b1","Type":"ContainerStarted","Data":"5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58"} Oct 31 12:53:11.402851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:11.402814 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" event={"ID":"90a85d43-3cca-48ff-b8a3-f8de2beff5b1","Type":"ContainerStarted","Data":"fccc3fb4d61266d487dbd43a12a6c916032f4333630fef0261aa99eaefbe4f84"} Oct 31 12:53:11.426914 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:11.426856 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podStartSLOduration=1.426841695 podStartE2EDuration="1.426841695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:11.423751346 +0000 UTC m=+1285.601469330" watchObservedRunningTime="2026-10-31 12:53:11.426841695 +0000 UTC m=+1285.604559681" Oct 31 12:53:11.826286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:11.826247 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:13.455474 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:13.455431 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:53:14.057569 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:14.057523 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:14.070592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:14.070560 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:14.435015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:14.434922 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:53:20.826337 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:20.826299 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:20.840133 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:20.840100 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:21.482269 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:21.482223 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:53:24.336959 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:24.336923 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:53:25.883703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:25.883662 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v"] Oct 31 12:53:25.890960 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:25.890927 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:25.909059 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:25.909028 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3dd1e5\"/\"premium-w2-3dd1e5-kserve-self-signed-certs\"" Oct 31 12:53:25.912663 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:25.912616 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v"] Oct 31 12:53:25.987543 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:53:25.987507 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:53:26.030188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.029716 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-tmp-dir\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.030188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.029814 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-home\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.030188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.029876 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qkf\" (UniqueName: \"kubernetes.io/projected/19c900e1-3941-45ce-b10f-d892634d7596-kube-api-access-s6qkf\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.030188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.029908 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c900e1-3941-45ce-b10f-d892634d7596-tls-certs\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.030188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.029934 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-model-cache\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.030188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.029988 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-dshm\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.131536 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.131499 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-tmp-dir\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.131776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.131561 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-home\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.131776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.131600 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qkf\" (UniqueName: \"kubernetes.io/projected/19c900e1-3941-45ce-b10f-d892634d7596-kube-api-access-s6qkf\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.131776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.131621 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c900e1-3941-45ce-b10f-d892634d7596-tls-certs\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.131776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.131678 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-model-cache\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.131776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.131727 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-dshm\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.132048 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.132009 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-tmp-dir\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.132107 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.132057 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-home\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.132241 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.132218 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-model-cache\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.134439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.134366 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-dshm\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.134699 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.134673 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c900e1-3941-45ce-b10f-d892634d7596-tls-certs\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.144131 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.144106 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qkf\" (UniqueName: \"kubernetes.io/projected/19c900e1-3941-45ce-b10f-d892634d7596-kube-api-access-s6qkf\") pod \"premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.215293 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.215251 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:26.495473 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:26.495446 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v"] Oct 31 12:53:26.497736 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:53:26.497702 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19c900e1_3941_45ce_b10f_d892634d7596.slice/crio-5a14b8f43dd771030a2dff9c4cadbb42734a57de67e4c4b2f475ef0adbd98351 WatchSource:0}: Error finding container 5a14b8f43dd771030a2dff9c4cadbb42734a57de67e4c4b2f475ef0adbd98351: Status 404 returned error can't find the container with id 5a14b8f43dd771030a2dff9c4cadbb42734a57de67e4c4b2f475ef0adbd98351 Oct 31 12:53:27.505834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:27.505789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" event={"ID":"19c900e1-3941-45ce-b10f-d892634d7596","Type":"ContainerStarted","Data":"e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72"} Oct 31 12:53:27.505834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:27.505847 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" event={"ID":"19c900e1-3941-45ce-b10f-d892634d7596","Type":"ContainerStarted","Data":"5a14b8f43dd771030a2dff9c4cadbb42734a57de67e4c4b2f475ef0adbd98351"} Oct 31 12:53:27.529258 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:27.529205 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podStartSLOduration=2.529190489 podStartE2EDuration="2.529190489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:27.526488173 +0000 UTC m=+1301.704206160" watchObservedRunningTime="2026-10-31 12:53:27.529190489 +0000 UTC m=+1301.706908473" Oct 31 12:53:28.216055 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:28.216023 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:31.378058 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.378016 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg"] Oct 31 12:53:31.382842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.382813 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.390112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.390084 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"kube-root-ca.crt\"" Oct 31 12:53:31.397328 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.397277 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"default-dockercfg-57z5d\"" Oct 31 12:53:31.397499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.397417 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"premium-w1-622e03-kserve-self-signed-certs\"" Oct 31 12:53:31.397499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.397441 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg"] Oct 31 12:53:31.489797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.489757 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-home\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.489989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.489860 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-model-cache\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.489989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.489934 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-dshm\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.489989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.489983 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d85f7d-a25a-48a3-81c4-08a485ba169d-tls-certs\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.490126 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.490024 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"openshift-service-ca.crt\"" Oct 31 12:53:31.490126 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.490048 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-tmp-dir\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.490126 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.490110 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9b6m\" (UniqueName: \"kubernetes.io/projected/88d85f7d-a25a-48a3-81c4-08a485ba169d-kube-api-access-d9b6m\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591093 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-model-cache\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591333 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591148 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-dshm\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591333 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591177 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d85f7d-a25a-48a3-81c4-08a485ba169d-tls-certs\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591333 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591208 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-tmp-dir\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591333 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591262 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9b6m\" (UniqueName: \"kubernetes.io/projected/88d85f7d-a25a-48a3-81c4-08a485ba169d-kube-api-access-d9b6m\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591333 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591302 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-home\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591597 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-model-cache\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591779 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591701 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-tmp-dir\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.591925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.591904 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-home\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.593586 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.593553 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-dshm\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.594329 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.594296 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d85f7d-a25a-48a3-81c4-08a485ba169d-tls-certs\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.604338 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.604315 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9b6m\" (UniqueName: \"kubernetes.io/projected/88d85f7d-a25a-48a3-81c4-08a485ba169d-kube-api-access-d9b6m\") pod \"premium-w1-622e03-kserve-557cbcbd4f-fc6mg\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:31.717360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:31.717330 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:32.005192 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:32.005163 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg"] Oct 31 12:53:32.007858 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:53:32.007821 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88d85f7d_a25a_48a3_81c4_08a485ba169d.slice/crio-6b0a4332b4cc6fd04bf8c9af0c8a8722972cd98f90302fe58176193fd2f59e45 WatchSource:0}: Error finding container 6b0a4332b4cc6fd04bf8c9af0c8a8722972cd98f90302fe58176193fd2f59e45: Status 404 returned error can't find the container with id 6b0a4332b4cc6fd04bf8c9af0c8a8722972cd98f90302fe58176193fd2f59e45 Oct 31 12:53:32.585426 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:32.585376 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" event={"ID":"88d85f7d-a25a-48a3-81c4-08a485ba169d","Type":"ContainerStarted","Data":"953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12"} Oct 31 12:53:32.585426 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:32.585430 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" event={"ID":"88d85f7d-a25a-48a3-81c4-08a485ba169d","Type":"ContainerStarted","Data":"6b0a4332b4cc6fd04bf8c9af0c8a8722972cd98f90302fe58176193fd2f59e45"} Oct 31 12:53:32.608190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:32.608132 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podStartSLOduration=1.608106032 podStartE2EDuration="1.608106032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:32.606762061 +0000 UTC m=+1306.784480045" watchObservedRunningTime="2026-10-31 12:53:32.608106032 +0000 UTC m=+1306.785824008" Oct 31 12:53:32.717827 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:32.717792 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:36.215574 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:36.215536 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:36.228459 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:36.228429 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:36.619923 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:36.619840 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:53:39.389362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:39.389322 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:53:41.718228 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:41.718192 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:41.731266 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:41.731239 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:42.657186 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:42.657155 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:53:50.469471 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.469436 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw"] Oct 31 12:53:50.474771 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.474747 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.482100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.482074 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:53:50.489670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.482360 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw"] Oct 31 12:53:50.489670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.482510 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:53:50.489670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.482793 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:53:50.489670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.482849 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:53:50.581098 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.581065 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9bhf\" (UniqueName: \"kubernetes.io/projected/d9694848-827c-4b7a-ab7c-643aeef809d5-kube-api-access-k9bhf\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.581282 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.581204 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.581325 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.581277 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.681913 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.681868 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.682131 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.681928 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9bhf\" (UniqueName: \"kubernetes.io/projected/d9694848-827c-4b7a-ab7c-643aeef809d5-kube-api-access-k9bhf\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.682131 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.682007 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.682581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.682554 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.684467 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.684441 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.695667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.695642 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9bhf\" (UniqueName: \"kubernetes.io/projected/d9694848-827c-4b7a-ab7c-643aeef809d5-kube-api-access-k9bhf\") pod \"maas-api-revoke-sub-8b56577a-dzhlw\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:50.791739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:50.791669 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:51.068913 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:51.068887 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw"] Oct 31 12:53:51.071527 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:53:51.071490 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9694848_827c_4b7a_ab7c_643aeef809d5.slice/crio-e26fe67c4f10e04d31facee2623f63491924036991710aa364d6befaaeaae8ba WatchSource:0}: Error finding container e26fe67c4f10e04d31facee2623f63491924036991710aa364d6befaaeaae8ba: Status 404 returned error can't find the container with id e26fe67c4f10e04d31facee2623f63491924036991710aa364d6befaaeaae8ba Oct 31 12:53:51.701962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:51.701865 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" event={"ID":"d9694848-827c-4b7a-ab7c-643aeef809d5","Type":"ContainerStarted","Data":"1f71ea3946aa8fcb1ad554798910e0efe1b2d8ce1bb09df8cbb569682a08e880"} Oct 31 12:53:51.701962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:51.701929 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" event={"ID":"d9694848-827c-4b7a-ab7c-643aeef809d5","Type":"ContainerStarted","Data":"e26fe67c4f10e04d31facee2623f63491924036991710aa364d6befaaeaae8ba"} Oct 31 12:53:51.721115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:51.721030 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" podStartSLOduration=1.721011287 podStartE2EDuration="1.721011287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:51.720268628 +0000 UTC m=+1325.897986612" watchObservedRunningTime="2026-10-31 12:53:51.721011287 +0000 UTC m=+1325.898729272" Oct 31 12:53:52.709844 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:52.709807 5074 generic.go:356] "Generic (PLEG): container finished" podID="d9694848-827c-4b7a-ab7c-643aeef809d5" containerID="1f71ea3946aa8fcb1ad554798910e0efe1b2d8ce1bb09df8cbb569682a08e880" exitCode=0 Oct 31 12:53:52.710265 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:52.709887 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" event={"ID":"d9694848-827c-4b7a-ab7c-643aeef809d5","Type":"ContainerDied","Data":"1f71ea3946aa8fcb1ad554798910e0efe1b2d8ce1bb09df8cbb569682a08e880"} Oct 31 12:53:53.860729 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:53.860698 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:53.916482 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:53.916440 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9694848-827c-4b7a-ab7c-643aeef809d5-kube-api-access-k9bhf\" (UniqueName: \"kubernetes.io/projected/d9694848-827c-4b7a-ab7c-643aeef809d5-kube-api-access-k9bhf\") pod \"d9694848-827c-4b7a-ab7c-643aeef809d5\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " Oct 31 12:53:53.916688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:53.916508 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-openshift-ca\") pod \"d9694848-827c-4b7a-ab7c-643aeef809d5\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " Oct 31 12:53:53.916688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:53.916658 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-xks-ca\") pod \"d9694848-827c-4b7a-ab7c-643aeef809d5\" (UID: \"d9694848-827c-4b7a-ab7c-643aeef809d5\") " Oct 31 12:53:53.917019 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:53.916986 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-openshift-ca" pod "d9694848-827c-4b7a-ab7c-643aeef809d5" (UID: "d9694848-827c-4b7a-ab7c-643aeef809d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:53:53.918752 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:53.918724 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9694848-827c-4b7a-ab7c-643aeef809d5-kube-api-access-k9bhf" pod "d9694848-827c-4b7a-ab7c-643aeef809d5" (UID: "d9694848-827c-4b7a-ab7c-643aeef809d5"). InnerVolumeSpecName "kube-api-access-k9bhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:53:53.918847 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:53.918790 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-xks-ca" pod "d9694848-827c-4b7a-ab7c-643aeef809d5" (UID: "d9694848-827c-4b7a-ab7c-643aeef809d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:53:54.018833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:54.018713 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:54.018995 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:54.018896 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9bhf\" (UniqueName: \"kubernetes.io/projected/d9694848-827c-4b7a-ab7c-643aeef809d5-kube-api-access-k9bhf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:54.018995 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:54.018920 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9694848-827c-4b7a-ab7c-643aeef809d5-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:53:54.693182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:54.693144 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:53:54.723391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:54.723353 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" event={"ID":"d9694848-827c-4b7a-ab7c-643aeef809d5","Type":"ContainerDied","Data":"e26fe67c4f10e04d31facee2623f63491924036991710aa364d6befaaeaae8ba"} Oct 31 12:53:54.723391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:54.723385 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw" Oct 31 12:53:54.723620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:53:54.723407 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e26fe67c4f10e04d31facee2623f63491924036991710aa364d6befaaeaae8ba" Oct 31 12:53:55.986752 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:53:55.986717 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:54:03.591429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.591384 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn"] Oct 31 12:54:03.592053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.591856 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d9694848-827c-4b7a-ab7c-643aeef809d5" containerName="revoke-keys" Oct 31 12:54:03.592053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.591868 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9694848-827c-4b7a-ab7c-643aeef809d5" containerName="revoke-keys" Oct 31 12:54:03.592053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.591989 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9694848-827c-4b7a-ab7c-643aeef809d5" containerName="revoke-keys" Oct 31 12:54:03.595118 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.595095 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.605036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.604570 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:54:03.605036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.604647 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:54:03.605036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.604737 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:54:03.605036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.604822 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:54:03.605476 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.605445 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn"] Oct 31 12:54:03.641676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.641648 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65"] Oct 31 12:54:03.645045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.645028 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.654396 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.654372 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65"] Oct 31 12:54:03.725939 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.725904 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93343735-2894-462b-a217-becdeddb3956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.726125 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.726020 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93343735-2894-462b-a217-becdeddb3956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.726125 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.726046 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.726125 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.726080 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxnw\" (UniqueName: \"kubernetes.io/projected/93343735-2894-462b-a217-becdeddb3956-kube-api-access-8fxnw\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.726235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.726176 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.726279 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.726242 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgjf\" (UniqueName: \"kubernetes.io/projected/75b32da8-638e-4150-bd3a-bb93ec713f9b-kube-api-access-wjgjf\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.827094 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.827061 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgjf\" (UniqueName: \"kubernetes.io/projected/75b32da8-638e-4150-bd3a-bb93ec713f9b-kube-api-access-wjgjf\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.827289 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.827162 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93343735-2894-462b-a217-becdeddb3956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.827289 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.827222 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93343735-2894-462b-a217-becdeddb3956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.827289 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.827244 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.827480 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.827287 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxnw\" (UniqueName: \"kubernetes.io/projected/93343735-2894-462b-a217-becdeddb3956-kube-api-access-8fxnw\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.827650 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.827597 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.828077 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.828049 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93343735-2894-462b-a217-becdeddb3956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.828420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.828396 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.829828 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.829805 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.829911 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.829811 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93343735-2894-462b-a217-becdeddb3956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.839445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.839424 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgjf\" (UniqueName: \"kubernetes.io/projected/75b32da8-638e-4150-bd3a-bb93ec713f9b-kube-api-access-wjgjf\") pod \"maas-api-revoke-sub-77626db3-l4fsn\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.839524 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.839461 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxnw\" (UniqueName: \"kubernetes.io/projected/93343735-2894-462b-a217-becdeddb3956-kube-api-access-8fxnw\") pod \"maas-api-revoke-sub-fef4914d-r9f65\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:03.912438 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.912357 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:03.962193 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:03.962154 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:04.201607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.201573 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn"] Oct 31 12:54:04.203155 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:54:04.203131 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b32da8_638e_4150_bd3a_bb93ec713f9b.slice/crio-b62b735a51662c6d5ef11133883049935f269ba69f03b4812d32466241074b9d WatchSource:0}: Error finding container b62b735a51662c6d5ef11133883049935f269ba69f03b4812d32466241074b9d: Status 404 returned error can't find the container with id b62b735a51662c6d5ef11133883049935f269ba69f03b4812d32466241074b9d Oct 31 12:54:04.270066 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.270019 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65"] Oct 31 12:54:04.271499 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:54:04.271474 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93343735_2894_462b_a217_becdeddb3956.slice/crio-3363dc5e7b24dc502b50db3085e583b8c0aac4106e8921565f40fce8b8feea3b WatchSource:0}: Error finding container 3363dc5e7b24dc502b50db3085e583b8c0aac4106e8921565f40fce8b8feea3b: Status 404 returned error can't find the container with id 3363dc5e7b24dc502b50db3085e583b8c0aac4106e8921565f40fce8b8feea3b Oct 31 12:54:04.273450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.273247 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955"] Oct 31 12:54:04.279070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.279042 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.284225 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.284167 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955"] Oct 31 12:54:04.433513 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.433478 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fpmg\" (UniqueName: \"kubernetes.io/projected/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-kube-api-access-9fpmg\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.433729 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.433667 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.433729 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.433721 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.534485 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.534456 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.534664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.534507 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.534664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.534558 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fpmg\" (UniqueName: \"kubernetes.io/projected/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-kube-api-access-9fpmg\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.535204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.535180 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.536849 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.536827 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.547796 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.547771 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fpmg\" (UniqueName: \"kubernetes.io/projected/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-kube-api-access-9fpmg\") pod \"maas-api-revoke-sub-b2846bc8-qm955\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.597518 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.597491 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:04.794306 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.794137 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" event={"ID":"93343735-2894-462b-a217-becdeddb3956","Type":"ContainerStarted","Data":"cfcdc8fd7aba5b309f22774ad447462d509de7fb591e06381273cf04d4094e2d"} Oct 31 12:54:04.794306 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.794206 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" event={"ID":"93343735-2894-462b-a217-becdeddb3956","Type":"ContainerStarted","Data":"3363dc5e7b24dc502b50db3085e583b8c0aac4106e8921565f40fce8b8feea3b"} Oct 31 12:54:04.796244 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.796214 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" event={"ID":"75b32da8-638e-4150-bd3a-bb93ec713f9b","Type":"ContainerStarted","Data":"1e244455942145df08ec43701db703f23de3c7bb8f2afe762442849a8ac6fd3e"} Oct 31 12:54:04.796372 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.796259 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" event={"ID":"75b32da8-638e-4150-bd3a-bb93ec713f9b","Type":"ContainerStarted","Data":"b62b735a51662c6d5ef11133883049935f269ba69f03b4812d32466241074b9d"} Oct 31 12:54:04.815784 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.815728 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" podStartSLOduration=1.815712552 podStartE2EDuration="1.815712552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:04.812725903 +0000 UTC m=+1338.990443887" watchObservedRunningTime="2026-10-31 12:54:04.815712552 +0000 UTC m=+1338.993430536" Oct 31 12:54:04.841331 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.841276 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" podStartSLOduration=1.8412589590000001 podStartE2EDuration="1.841258959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:04.837766148 +0000 UTC m=+1339.015484137" watchObservedRunningTime="2026-10-31 12:54:04.841258959 +0000 UTC m=+1339.018976945" Oct 31 12:54:04.878090 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:04.878060 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955"] Oct 31 12:54:04.879582 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:54:04.879554 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5dd1ff_e593_4d4b_9730_f879c6f6f226.slice/crio-4c204a266b46ed97baba85e4b913b291c9c8361f915a6d6470892bae9b5c380e WatchSource:0}: Error finding container 4c204a266b46ed97baba85e4b913b291c9c8361f915a6d6470892bae9b5c380e: Status 404 returned error can't find the container with id 4c204a266b46ed97baba85e4b913b291c9c8361f915a6d6470892bae9b5c380e Oct 31 12:54:05.803089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:05.803054 5074 generic.go:356] "Generic (PLEG): container finished" podID="75b32da8-638e-4150-bd3a-bb93ec713f9b" containerID="1e244455942145df08ec43701db703f23de3c7bb8f2afe762442849a8ac6fd3e" exitCode=0 Oct 31 12:54:05.803559 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:05.803142 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" event={"ID":"75b32da8-638e-4150-bd3a-bb93ec713f9b","Type":"ContainerDied","Data":"1e244455942145df08ec43701db703f23de3c7bb8f2afe762442849a8ac6fd3e"} Oct 31 12:54:05.804904 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:05.804878 5074 generic.go:356] "Generic (PLEG): container finished" podID="93343735-2894-462b-a217-becdeddb3956" containerID="cfcdc8fd7aba5b309f22774ad447462d509de7fb591e06381273cf04d4094e2d" exitCode=0 Oct 31 12:54:05.805053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:05.804904 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" event={"ID":"93343735-2894-462b-a217-becdeddb3956","Type":"ContainerDied","Data":"cfcdc8fd7aba5b309f22774ad447462d509de7fb591e06381273cf04d4094e2d"} Oct 31 12:54:05.806622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:05.806599 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" event={"ID":"5a5dd1ff-e593-4d4b-9730-f879c6f6f226","Type":"ContainerStarted","Data":"0157079f3a1605c8e137e67b01db7efd498f4acf279d278bb045f2edf0d1320f"} Oct 31 12:54:05.806799 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:05.806651 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" event={"ID":"5a5dd1ff-e593-4d4b-9730-f879c6f6f226","Type":"ContainerStarted","Data":"4c204a266b46ed97baba85e4b913b291c9c8361f915a6d6470892bae9b5c380e"} Oct 31 12:54:05.843944 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:05.843885 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" podStartSLOduration=1.843874059 podStartE2EDuration="1.843874059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:05.841984155 +0000 UTC m=+1340.019702141" watchObservedRunningTime="2026-10-31 12:54:05.843874059 +0000 UTC m=+1340.021592042" Oct 31 12:54:06.815063 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:06.815028 5074 generic.go:356] "Generic (PLEG): container finished" podID="5a5dd1ff-e593-4d4b-9730-f879c6f6f226" containerID="0157079f3a1605c8e137e67b01db7efd498f4acf279d278bb045f2edf0d1320f" exitCode=0 Oct 31 12:54:06.815544 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:06.815114 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" event={"ID":"5a5dd1ff-e593-4d4b-9730-f879c6f6f226","Type":"ContainerDied","Data":"0157079f3a1605c8e137e67b01db7efd498f4acf279d278bb045f2edf0d1320f"} Oct 31 12:54:06.994244 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:06.994219 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:07.027242 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.027207 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:07.063214 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.063178 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-openshift-ca\") pod \"75b32da8-638e-4150-bd3a-bb93ec713f9b\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " Oct 31 12:54:07.063403 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.063382 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-xks-ca\") pod \"75b32da8-638e-4150-bd3a-bb93ec713f9b\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " Oct 31 12:54:07.063539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.063440 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b32da8-638e-4150-bd3a-bb93ec713f9b-kube-api-access-wjgjf\" (UniqueName: \"kubernetes.io/projected/75b32da8-638e-4150-bd3a-bb93ec713f9b-kube-api-access-wjgjf\") pod \"75b32da8-638e-4150-bd3a-bb93ec713f9b\" (UID: \"75b32da8-638e-4150-bd3a-bb93ec713f9b\") " Oct 31 12:54:07.063539 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.063504 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-openshift-ca" pod "75b32da8-638e-4150-bd3a-bb93ec713f9b" (UID: "75b32da8-638e-4150-bd3a-bb93ec713f9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:07.063929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.063901 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:07.065453 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.065395 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-xks-ca" pod "75b32da8-638e-4150-bd3a-bb93ec713f9b" (UID: "75b32da8-638e-4150-bd3a-bb93ec713f9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:07.065592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.065567 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b32da8-638e-4150-bd3a-bb93ec713f9b-kube-api-access-wjgjf" pod "75b32da8-638e-4150-bd3a-bb93ec713f9b" (UID: "75b32da8-638e-4150-bd3a-bb93ec713f9b"). InnerVolumeSpecName "kube-api-access-wjgjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:07.164419 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.164390 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93343735-2894-462b-a217-becdeddb3956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93343735-2894-462b-a217-becdeddb3956-maas-api-xks-ca\") pod \"93343735-2894-462b-a217-becdeddb3956\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " Oct 31 12:54:07.164584 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.164452 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93343735-2894-462b-a217-becdeddb3956-kube-api-access-8fxnw\" (UniqueName: \"kubernetes.io/projected/93343735-2894-462b-a217-becdeddb3956-kube-api-access-8fxnw\") pod \"93343735-2894-462b-a217-becdeddb3956\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " Oct 31 12:54:07.164584 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.164474 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93343735-2894-462b-a217-becdeddb3956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93343735-2894-462b-a217-becdeddb3956-maas-api-openshift-ca\") pod \"93343735-2894-462b-a217-becdeddb3956\" (UID: \"93343735-2894-462b-a217-becdeddb3956\") " Oct 31 12:54:07.164769 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.164753 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjgjf\" (UniqueName: \"kubernetes.io/projected/75b32da8-638e-4150-bd3a-bb93ec713f9b-kube-api-access-wjgjf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:07.164835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.164771 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b32da8-638e-4150-bd3a-bb93ec713f9b-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:07.164948 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.164916 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93343735-2894-462b-a217-becdeddb3956-maas-api-openshift-ca" pod "93343735-2894-462b-a217-becdeddb3956" (UID: "93343735-2894-462b-a217-becdeddb3956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:07.166482 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.166458 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93343735-2894-462b-a217-becdeddb3956-kube-api-access-8fxnw" pod "93343735-2894-462b-a217-becdeddb3956" (UID: "93343735-2894-462b-a217-becdeddb3956"). InnerVolumeSpecName "kube-api-access-8fxnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:07.166583 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.166465 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93343735-2894-462b-a217-becdeddb3956-maas-api-xks-ca" pod "93343735-2894-462b-a217-becdeddb3956" (UID: "93343735-2894-462b-a217-becdeddb3956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:07.265510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.265478 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93343735-2894-462b-a217-becdeddb3956-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:07.265510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.265511 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxnw\" (UniqueName: \"kubernetes.io/projected/93343735-2894-462b-a217-becdeddb3956-kube-api-access-8fxnw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:07.265766 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.265525 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93343735-2894-462b-a217-becdeddb3956-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:07.821699 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.821668 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" Oct 31 12:54:07.822156 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.821693 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65" event={"ID":"93343735-2894-462b-a217-becdeddb3956","Type":"ContainerDied","Data":"3363dc5e7b24dc502b50db3085e583b8c0aac4106e8921565f40fce8b8feea3b"} Oct 31 12:54:07.822156 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.821750 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3363dc5e7b24dc502b50db3085e583b8c0aac4106e8921565f40fce8b8feea3b" Oct 31 12:54:07.823148 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.823122 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" Oct 31 12:54:07.823311 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.823149 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn" event={"ID":"75b32da8-638e-4150-bd3a-bb93ec713f9b","Type":"ContainerDied","Data":"b62b735a51662c6d5ef11133883049935f269ba69f03b4812d32466241074b9d"} Oct 31 12:54:07.823311 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.823192 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b62b735a51662c6d5ef11133883049935f269ba69f03b4812d32466241074b9d" Oct 31 12:54:07.955518 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:07.955495 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:08.074124 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.074033 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-kube-api-access-9fpmg\" (UniqueName: \"kubernetes.io/projected/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-kube-api-access-9fpmg\") pod \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " Oct 31 12:54:08.074124 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.074120 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-openshift-ca\") pod \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " Oct 31 12:54:08.074303 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.074172 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-xks-ca\") pod \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\" (UID: \"5a5dd1ff-e593-4d4b-9730-f879c6f6f226\") " Oct 31 12:54:08.074803 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.074763 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-openshift-ca" pod "5a5dd1ff-e593-4d4b-9730-f879c6f6f226" (UID: "5a5dd1ff-e593-4d4b-9730-f879c6f6f226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:08.076796 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.076760 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-xks-ca" pod "5a5dd1ff-e593-4d4b-9730-f879c6f6f226" (UID: "5a5dd1ff-e593-4d4b-9730-f879c6f6f226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:08.076907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.076801 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-kube-api-access-9fpmg" pod "5a5dd1ff-e593-4d4b-9730-f879c6f6f226" (UID: "5a5dd1ff-e593-4d4b-9730-f879c6f6f226"). InnerVolumeSpecName "kube-api-access-9fpmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:08.175623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.175590 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:08.175623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.175622 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fpmg\" (UniqueName: \"kubernetes.io/projected/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-kube-api-access-9fpmg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:08.175820 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.175648 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5dd1ff-e593-4d4b-9730-f879c6f6f226-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:08.830562 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.830523 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" Oct 31 12:54:08.830562 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.830548 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955" event={"ID":"5a5dd1ff-e593-4d4b-9730-f879c6f6f226","Type":"ContainerDied","Data":"4c204a266b46ed97baba85e4b913b291c9c8361f915a6d6470892bae9b5c380e"} Oct 31 12:54:08.831032 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:08.830604 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c204a266b46ed97baba85e4b913b291c9c8361f915a6d6470892bae9b5c380e" Oct 31 12:54:17.932741 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.931591 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc"] Oct 31 12:54:17.932741 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.932588 5074 cpu_manager.go:414] "containerMap: removing container" podUID="93343735-2894-462b-a217-becdeddb3956" containerName="revoke-keys" Oct 31 12:54:17.932741 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.932613 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93343735-2894-462b-a217-becdeddb3956" containerName="revoke-keys" Oct 31 12:54:17.934812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.932627 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5a5dd1ff-e593-4d4b-9730-f879c6f6f226" containerName="revoke-keys" Oct 31 12:54:17.934812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.933372 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5dd1ff-e593-4d4b-9730-f879c6f6f226" containerName="revoke-keys" Oct 31 12:54:17.934812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.933472 5074 cpu_manager.go:414] "containerMap: removing container" podUID="75b32da8-638e-4150-bd3a-bb93ec713f9b" containerName="revoke-keys" Oct 31 12:54:17.934812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.933481 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b32da8-638e-4150-bd3a-bb93ec713f9b" containerName="revoke-keys" Oct 31 12:54:17.934812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.933770 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="93343735-2894-462b-a217-becdeddb3956" containerName="revoke-keys" Oct 31 12:54:17.934812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.933786 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5dd1ff-e593-4d4b-9730-f879c6f6f226" containerName="revoke-keys" Oct 31 12:54:17.934812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.933813 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b32da8-638e-4150-bd3a-bb93ec713f9b" containerName="revoke-keys" Oct 31 12:54:17.940658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.940619 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:17.944929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.944908 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:54:17.945010 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.944931 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:54:17.946247 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.946223 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:54:17.946247 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.946225 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:54:17.952073 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.952051 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc"] Oct 31 12:54:17.993920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.993881 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscmq\" (UniqueName: \"kubernetes.io/projected/9c1997fa-4ab2-4892-ad00-48e874340162-kube-api-access-fscmq\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:17.994138 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.993937 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:17.994138 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:17.994106 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.094670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.094624 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fscmq\" (UniqueName: \"kubernetes.io/projected/9c1997fa-4ab2-4892-ad00-48e874340162-kube-api-access-fscmq\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.094855 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.094678 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.094855 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.094748 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.095545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.095506 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.097145 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.097121 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.109039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.109014 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscmq\" (UniqueName: \"kubernetes.io/projected/9c1997fa-4ab2-4892-ad00-48e874340162-kube-api-access-fscmq\") pod \"maas-api-revoke-sub-5c9bc428-xtsnc\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.215142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.215107 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt"] Oct 31 12:54:18.219881 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.219860 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.226753 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.226724 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt"] Oct 31 12:54:18.255662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.255614 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:18.296975 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.296937 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.296975 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.296981 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cns4c\" (UniqueName: \"kubernetes.io/projected/44323960-f9ed-489c-8327-7ed50a173eab-kube-api-access-cns4c\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.297260 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.297178 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.398175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.398138 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.398357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.398291 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.398357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.398315 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cns4c\" (UniqueName: \"kubernetes.io/projected/44323960-f9ed-489c-8327-7ed50a173eab-kube-api-access-cns4c\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.399112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.399076 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.400841 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.400818 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.430985 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.430914 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cns4c\" (UniqueName: \"kubernetes.io/projected/44323960-f9ed-489c-8327-7ed50a173eab-kube-api-access-cns4c\") pod \"maas-api-revoke-sub-16041536-6dzmt\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.537003 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.536911 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:18.549309 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.549283 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc"] Oct 31 12:54:18.552125 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:54:18.552091 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1997fa_4ab2_4892_ad00_48e874340162.slice/crio-ce4bbf261c3088ccc51d707d20f16685dd110c702df819d67bd9176d570516a8 WatchSource:0}: Error finding container ce4bbf261c3088ccc51d707d20f16685dd110c702df819d67bd9176d570516a8: Status 404 returned error can't find the container with id ce4bbf261c3088ccc51d707d20f16685dd110c702df819d67bd9176d570516a8 Oct 31 12:54:18.827380 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.827342 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt"] Oct 31 12:54:18.828329 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:54:18.828292 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44323960_f9ed_489c_8327_7ed50a173eab.slice/crio-1f56e8fa9d5e8a13d9fb280536ad5e63eeacace76facaab1d3c2c0ed59fbf0a1 WatchSource:0}: Error finding container 1f56e8fa9d5e8a13d9fb280536ad5e63eeacace76facaab1d3c2c0ed59fbf0a1: Status 404 returned error can't find the container with id 1f56e8fa9d5e8a13d9fb280536ad5e63eeacace76facaab1d3c2c0ed59fbf0a1 Oct 31 12:54:18.915978 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.915934 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" event={"ID":"44323960-f9ed-489c-8327-7ed50a173eab","Type":"ContainerStarted","Data":"1f56e8fa9d5e8a13d9fb280536ad5e63eeacace76facaab1d3c2c0ed59fbf0a1"} Oct 31 12:54:18.917985 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.917945 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" event={"ID":"9c1997fa-4ab2-4892-ad00-48e874340162","Type":"ContainerStarted","Data":"ae1bb5c20060d97c78da9912b6d64bf451ce35debf8d17cae4e534a24f6557ea"} Oct 31 12:54:18.918126 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.918004 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" event={"ID":"9c1997fa-4ab2-4892-ad00-48e874340162","Type":"ContainerStarted","Data":"ce4bbf261c3088ccc51d707d20f16685dd110c702df819d67bd9176d570516a8"} Oct 31 12:54:18.939433 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:18.939374 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" podStartSLOduration=1.939358263 podStartE2EDuration="1.939358263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:18.935470671 +0000 UTC m=+1353.113188656" watchObservedRunningTime="2026-10-31 12:54:18.939358263 +0000 UTC m=+1353.117076254" Oct 31 12:54:19.198308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.198277 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:54:19.851123 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.851078 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp"] Oct 31 12:54:19.855126 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.855097 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:19.869378 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.869158 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp"] Oct 31 12:54:19.918963 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.918916 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:19.919171 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.919105 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:19.919171 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.919143 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gktl\" (UniqueName: \"kubernetes.io/projected/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-kube-api-access-2gktl\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:19.940725 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.933066 5074 generic.go:356] "Generic (PLEG): container finished" podID="9c1997fa-4ab2-4892-ad00-48e874340162" containerID="ae1bb5c20060d97c78da9912b6d64bf451ce35debf8d17cae4e534a24f6557ea" exitCode=0 Oct 31 12:54:19.940725 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.935552 5074 generic.go:356] "Generic (PLEG): container finished" podID="44323960-f9ed-489c-8327-7ed50a173eab" containerID="8daba1a8dbda807d81ffbb46d5f7f0730b16f9100fadf4558b811941ae50e10e" exitCode=0 Oct 31 12:54:19.947168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.946200 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" event={"ID":"9c1997fa-4ab2-4892-ad00-48e874340162","Type":"ContainerDied","Data":"ae1bb5c20060d97c78da9912b6d64bf451ce35debf8d17cae4e534a24f6557ea"} Oct 31 12:54:19.947168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:19.946267 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" event={"ID":"44323960-f9ed-489c-8327-7ed50a173eab","Type":"ContainerDied","Data":"8daba1a8dbda807d81ffbb46d5f7f0730b16f9100fadf4558b811941ae50e10e"} Oct 31 12:54:20.020653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.020545 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gktl\" (UniqueName: \"kubernetes.io/projected/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-kube-api-access-2gktl\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:20.020821 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.020705 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:20.025667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.022216 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:20.025667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.023210 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:20.034045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.033995 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:20.035936 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.035900 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gktl\" (UniqueName: \"kubernetes.io/projected/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-kube-api-access-2gktl\") pod \"maas-api-revoke-sub-8b20b56c-2qnzp\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:20.169063 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.169020 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:20.519506 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.519466 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp"] Oct 31 12:54:20.958776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.958737 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" event={"ID":"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8","Type":"ContainerStarted","Data":"395b9906a3cb4019d12ec66091233c1b826a6ae2a87dcaa57f41f77443d0ec85"} Oct 31 12:54:20.959356 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.959333 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" event={"ID":"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8","Type":"ContainerStarted","Data":"94d7a4a7d573f902fd8892177daf82979630765551c9b5a9af2f707b06c40fd2"} Oct 31 12:54:20.985201 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:20.985120 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" podStartSLOduration=1.985101354 podStartE2EDuration="1.985101354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:20.984570445 +0000 UTC m=+1355.162288429" watchObservedRunningTime="2026-10-31 12:54:20.985101354 +0000 UTC m=+1355.162819341" Oct 31 12:54:21.570160 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.569829 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:21.757737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.757695 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-xks-ca\") pod \"9c1997fa-4ab2-4892-ad00-48e874340162\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " Oct 31 12:54:21.770856 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.770821 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-openshift-ca\") pod \"9c1997fa-4ab2-4892-ad00-48e874340162\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " Oct 31 12:54:21.781658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.777724 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1997fa-4ab2-4892-ad00-48e874340162-kube-api-access-fscmq\" (UniqueName: \"kubernetes.io/projected/9c1997fa-4ab2-4892-ad00-48e874340162-kube-api-access-fscmq\") pod \"9c1997fa-4ab2-4892-ad00-48e874340162\" (UID: \"9c1997fa-4ab2-4892-ad00-48e874340162\") " Oct 31 12:54:21.832689 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.774075 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-openshift-ca" pod "9c1997fa-4ab2-4892-ad00-48e874340162" (UID: "9c1997fa-4ab2-4892-ad00-48e874340162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:21.832952 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.822036 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:21.846667 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:54:21.845835 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded772b52_b9a0_438b_b062_d6e0eaeb5aa8.slice/crio-conmon-395b9906a3cb4019d12ec66091233c1b826a6ae2a87dcaa57f41f77443d0ec85.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded772b52_b9a0_438b_b062_d6e0eaeb5aa8.slice/crio-395b9906a3cb4019d12ec66091233c1b826a6ae2a87dcaa57f41f77443d0ec85.scope\": RecentStats: unable to find data in memory cache]" Oct 31 12:54:21.911708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.910727 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-xks-ca\") pod \"44323960-f9ed-489c-8327-7ed50a173eab\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " Oct 31 12:54:21.911708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.910845 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-openshift-ca\") pod \"44323960-f9ed-489c-8327-7ed50a173eab\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " Oct 31 12:54:21.911708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.910880 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44323960-f9ed-489c-8327-7ed50a173eab-kube-api-access-cns4c\" (UniqueName: \"kubernetes.io/projected/44323960-f9ed-489c-8327-7ed50a173eab-kube-api-access-cns4c\") pod \"44323960-f9ed-489c-8327-7ed50a173eab\" (UID: \"44323960-f9ed-489c-8327-7ed50a173eab\") " Oct 31 12:54:21.911708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.911263 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:21.913761 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.913735 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-openshift-ca" pod "44323960-f9ed-489c-8327-7ed50a173eab" (UID: "44323960-f9ed-489c-8327-7ed50a173eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:21.922680 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.921732 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-xks-ca" pod "9c1997fa-4ab2-4892-ad00-48e874340162" (UID: "9c1997fa-4ab2-4892-ad00-48e874340162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:21.993653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:21.968598 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-xks-ca" pod "44323960-f9ed-489c-8327-7ed50a173eab" (UID: "44323960-f9ed-489c-8327-7ed50a173eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:22.043776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.043678 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1997fa-4ab2-4892-ad00-48e874340162-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:22.043776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.043717 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:22.043776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.043733 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44323960-f9ed-489c-8327-7ed50a173eab-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:22.059767 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.058678 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44323960-f9ed-489c-8327-7ed50a173eab-kube-api-access-cns4c" pod "44323960-f9ed-489c-8327-7ed50a173eab" (UID: "44323960-f9ed-489c-8327-7ed50a173eab"). InnerVolumeSpecName "kube-api-access-cns4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:22.066450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.066223 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1997fa-4ab2-4892-ad00-48e874340162-kube-api-access-fscmq" pod "9c1997fa-4ab2-4892-ad00-48e874340162" (UID: "9c1997fa-4ab2-4892-ad00-48e874340162"). InnerVolumeSpecName "kube-api-access-fscmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:22.087679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.067138 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" event={"ID":"9c1997fa-4ab2-4892-ad00-48e874340162","Type":"ContainerDied","Data":"ce4bbf261c3088ccc51d707d20f16685dd110c702df819d67bd9176d570516a8"} Oct 31 12:54:22.087679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.067178 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce4bbf261c3088ccc51d707d20f16685dd110c702df819d67bd9176d570516a8" Oct 31 12:54:22.087679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.067241 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc" Oct 31 12:54:22.118681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.101600 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" event={"ID":"44323960-f9ed-489c-8327-7ed50a173eab","Type":"ContainerDied","Data":"1f56e8fa9d5e8a13d9fb280536ad5e63eeacace76facaab1d3c2c0ed59fbf0a1"} Oct 31 12:54:22.118681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.101687 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f56e8fa9d5e8a13d9fb280536ad5e63eeacace76facaab1d3c2c0ed59fbf0a1" Oct 31 12:54:22.118681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.101758 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt" Oct 31 12:54:22.134680 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.124030 5074 generic.go:356] "Generic (PLEG): container finished" podID="ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" containerID="395b9906a3cb4019d12ec66091233c1b826a6ae2a87dcaa57f41f77443d0ec85" exitCode=0 Oct 31 12:54:22.134680 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.124156 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" event={"ID":"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8","Type":"ContainerDied","Data":"395b9906a3cb4019d12ec66091233c1b826a6ae2a87dcaa57f41f77443d0ec85"} Oct 31 12:54:22.162715 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.153494 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cns4c\" (UniqueName: \"kubernetes.io/projected/44323960-f9ed-489c-8327-7ed50a173eab-kube-api-access-cns4c\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:22.162715 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:22.153533 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fscmq\" (UniqueName: \"kubernetes.io/projected/9c1997fa-4ab2-4892-ad00-48e874340162-kube-api-access-fscmq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:23.284497 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.284467 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:23.380184 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.380153 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-xks-ca\") pod \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " Oct 31 12:54:23.380358 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.380310 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-openshift-ca\") pod \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " Oct 31 12:54:23.380424 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.380378 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-kube-api-access-2gktl\" (UniqueName: \"kubernetes.io/projected/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-kube-api-access-2gktl\") pod \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\" (UID: \"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8\") " Oct 31 12:54:23.380695 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.380665 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-openshift-ca" pod "ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" (UID: "ed772b52-b9a0-438b-b062-d6e0eaeb5aa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:23.380827 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.380767 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:23.382439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.382411 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-xks-ca" pod "ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" (UID: "ed772b52-b9a0-438b-b062-d6e0eaeb5aa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:23.382610 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.382586 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-kube-api-access-2gktl" pod "ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" (UID: "ed772b52-b9a0-438b-b062-d6e0eaeb5aa8"). InnerVolumeSpecName "kube-api-access-2gktl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:23.481360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.481326 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:23.481360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:23.481358 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gktl\" (UniqueName: \"kubernetes.io/projected/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8-kube-api-access-2gktl\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:24.147350 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:24.147318 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" Oct 31 12:54:24.147350 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:24.147335 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp" event={"ID":"ed772b52-b9a0-438b-b062-d6e0eaeb5aa8","Type":"ContainerDied","Data":"94d7a4a7d573f902fd8892177daf82979630765551c9b5a9af2f707b06c40fd2"} Oct 31 12:54:24.147602 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:24.147401 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94d7a4a7d573f902fd8892177daf82979630765551c9b5a9af2f707b06c40fd2" Oct 31 12:54:25.984807 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:54:25.984770 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:54:28.599650 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.599468 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z"] Oct 31 12:54:28.600199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600176 5074 cpu_manager.go:414] "containerMap: removing container" podUID="44323960-f9ed-489c-8327-7ed50a173eab" containerName="revoke-keys" Oct 31 12:54:28.600261 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600201 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44323960-f9ed-489c-8327-7ed50a173eab" containerName="revoke-keys" Oct 31 12:54:28.600261 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600250 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9c1997fa-4ab2-4892-ad00-48e874340162" containerName="revoke-keys" Oct 31 12:54:28.600334 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600259 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1997fa-4ab2-4892-ad00-48e874340162" containerName="revoke-keys" Oct 31 12:54:28.600334 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600299 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" containerName="revoke-keys" Oct 31 12:54:28.600334 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600308 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" containerName="revoke-keys" Oct 31 12:54:28.600466 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600452 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1997fa-4ab2-4892-ad00-48e874340162" containerName="revoke-keys" Oct 31 12:54:28.600514 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600475 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" containerName="revoke-keys" Oct 31 12:54:28.600514 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.600487 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="44323960-f9ed-489c-8327-7ed50a173eab" containerName="revoke-keys" Oct 31 12:54:28.604502 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.604479 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.618563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.618543 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z"] Oct 31 12:54:28.736405 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736354 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.736612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736423 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.736612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736497 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ef088b-8828-4966-9457-fe4346c638b1-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.736612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736533 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.736612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736586 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ef088b-8828-4966-9457-fe4346c638b1-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.736873 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736675 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.736873 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736744 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.736979 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736933 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5qh\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-kube-api-access-kl5qh\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.737037 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.736979 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838252 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5qh\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-kube-api-access-kl5qh\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838309 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838369 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838404 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838453 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ef088b-8828-4966-9457-fe4346c638b1-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838488 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838541 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ef088b-8828-4966-9457-fe4346c638b1-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838613 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.838658 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.839324 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.841683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.840118 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.843660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.842792 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ef088b-8828-4966-9457-fe4346c638b1-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.843660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.843056 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.852691 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.852079 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.852691 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.852407 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.858461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.855370 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5qh\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-kube-api-access-kl5qh\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.858461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.857880 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.861249 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.861217 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ef088b-8828-4966-9457-fe4346c638b1-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:28.933083 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:28.933033 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:29.273661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:29.270290 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z"] Oct 31 12:54:29.285818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:29.285778 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:29.286312 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:29.286286 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:29.286483 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:29.286460 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:30.097665 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.097614 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn"] Oct 31 12:54:30.291665 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.288913 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" event={"ID":"c6ef088b-8828-4966-9457-fe4346c638b1","Type":"ContainerStarted","Data":"3cd3da5904addaf8e3e784eba2d8aa0b4dbd99436a3d403339de23f1a94b83bd"} Oct 31 12:54:30.291665 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.288981 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" event={"ID":"c6ef088b-8828-4966-9457-fe4346c638b1","Type":"ContainerStarted","Data":"21d29c7af0c064f9980fba8f8423bac48815d513c511f1fad75fcab06e0049ca"} Oct 31 12:54:30.291665 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.289013 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn"] Oct 31 12:54:30.291665 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.289129 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.312672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.306351 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:54:30.312672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.307306 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:54:30.312672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.307399 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:54:30.312672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.307470 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:54:30.395209 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.395087 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.395536 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.395429 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.395710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.395690 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnlg2\" (UniqueName: \"kubernetes.io/projected/cc782c36-7587-4c5c-b8cb-435c18b86e2c-kube-api-access-mnlg2\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.439658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.423608 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podStartSLOduration=2.423587091 podStartE2EDuration="2.423587091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:30.379262574 +0000 UTC m=+1364.556980559" watchObservedRunningTime="2026-10-31 12:54:30.423587091 +0000 UTC m=+1364.601305074" Oct 31 12:54:30.498516 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.498460 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.498748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.498528 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.498748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.498552 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnlg2\" (UniqueName: \"kubernetes.io/projected/cc782c36-7587-4c5c-b8cb-435c18b86e2c-kube-api-access-mnlg2\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.499838 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.499811 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.530228 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.530180 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnlg2\" (UniqueName: \"kubernetes.io/projected/cc782c36-7587-4c5c-b8cb-435c18b86e2c-kube-api-access-mnlg2\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.561367 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.561321 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11b044ea-52qhn\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.766074 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.766042 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:30.946684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.946573 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:30.947026 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.946999 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:54:30.947235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:30.947208 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 12:54:31.694667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:31.685703 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn"] Oct 31 12:54:31.829666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:31.828731 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:54:31.973704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:31.959612 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:54:32.027653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:32.027063 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 12:54:32.418658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:32.410570 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" event={"ID":"cc782c36-7587-4c5c-b8cb-435c18b86e2c","Type":"ContainerStarted","Data":"df9a4ec1a1417310a754fca53193b28555d4176bae90ebbcdb6905c765baaeca"} Oct 31 12:54:32.418658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:32.410684 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" event={"ID":"cc782c36-7587-4c5c-b8cb-435c18b86e2c","Type":"ContainerStarted","Data":"58aff1fe08ec87b37feb4a9fb8c1936908ec4fc3cca205bfb2d3c687eaa64b2e"} Oct 31 12:54:32.933700 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:32.933656 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:54:32.934204 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:32.933719 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 12:54:33.455628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:33.451447 5074 generic.go:356] "Generic (PLEG): container finished" podID="cc782c36-7587-4c5c-b8cb-435c18b86e2c" containerID="df9a4ec1a1417310a754fca53193b28555d4176bae90ebbcdb6905c765baaeca" exitCode=0 Oct 31 12:54:33.455628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:33.451594 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" event={"ID":"cc782c36-7587-4c5c-b8cb-435c18b86e2c","Type":"ContainerDied","Data":"df9a4ec1a1417310a754fca53193b28555d4176bae90ebbcdb6905c765baaeca"} Oct 31 12:54:33.933492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:33.933455 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:54:33.933690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:33.933504 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 12:54:34.730507 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.730484 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:34.743575 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.743550 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-xks-ca\") pod \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " Oct 31 12:54:34.743687 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.743672 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc782c36-7587-4c5c-b8cb-435c18b86e2c-kube-api-access-mnlg2\" (UniqueName: \"kubernetes.io/projected/cc782c36-7587-4c5c-b8cb-435c18b86e2c-kube-api-access-mnlg2\") pod \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " Oct 31 12:54:34.743772 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.743724 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-openshift-ca\") pod \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\" (UID: \"cc782c36-7587-4c5c-b8cb-435c18b86e2c\") " Oct 31 12:54:34.744289 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.744262 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-openshift-ca" pod "cc782c36-7587-4c5c-b8cb-435c18b86e2c" (UID: "cc782c36-7587-4c5c-b8cb-435c18b86e2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:54:34.745509 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.745482 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-xks-ca" pod "cc782c36-7587-4c5c-b8cb-435c18b86e2c" (UID: "cc782c36-7587-4c5c-b8cb-435c18b86e2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:54:34.745725 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.745706 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc782c36-7587-4c5c-b8cb-435c18b86e2c-kube-api-access-mnlg2" pod "cc782c36-7587-4c5c-b8cb-435c18b86e2c" (UID: "cc782c36-7587-4c5c-b8cb-435c18b86e2c"). InnerVolumeSpecName "kube-api-access-mnlg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:54:34.845115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.845077 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.845115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.845110 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc782c36-7587-4c5c-b8cb-435c18b86e2c-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.845115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.845120 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnlg2\" (UniqueName: \"kubernetes.io/projected/cc782c36-7587-4c5c-b8cb-435c18b86e2c-kube-api-access-mnlg2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:54:34.934230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.934197 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:54:34.934400 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:34.934258 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 12:54:35.480497 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:35.480390 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" event={"ID":"cc782c36-7587-4c5c-b8cb-435c18b86e2c","Type":"ContainerDied","Data":"58aff1fe08ec87b37feb4a9fb8c1936908ec4fc3cca205bfb2d3c687eaa64b2e"} Oct 31 12:54:35.480835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:35.480474 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58aff1fe08ec87b37feb4a9fb8c1936908ec4fc3cca205bfb2d3c687eaa64b2e" Oct 31 12:54:35.481025 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:35.480983 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn" Oct 31 12:54:35.933716 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:35.933613 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 12:54:35.933716 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:35.933693 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 12:54:36.090625 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.090583 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:54:36.889501 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.889461 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c"] Oct 31 12:54:36.890236 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.890215 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cc782c36-7587-4c5c-b8cb-435c18b86e2c" containerName="revoke-keys" Oct 31 12:54:36.890300 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.890239 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc782c36-7587-4c5c-b8cb-435c18b86e2c" containerName="revoke-keys" Oct 31 12:54:36.890373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.890362 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc782c36-7587-4c5c-b8cb-435c18b86e2c" containerName="revoke-keys" Oct 31 12:54:36.894478 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.894444 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.902220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.902193 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7ae9e7-openshift-default-dockercfg-pplr5\"" Oct 31 12:54:36.907910 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.907875 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c"] Oct 31 12:54:36.938602 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.938529 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:36.939017 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.938972 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:36.939775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.939744 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 12:54:36.970050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970017 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970209 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970053 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970209 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970140 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970244 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970281 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970315 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970472 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970456 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970529 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970505 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:36.970578 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:36.970528 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgnk\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-kube-api-access-gvgnk\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.071491 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071446 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.071491 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071493 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.071777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071525 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.071777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071561 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.071777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071593 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.071777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071625 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.071777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071734 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.072057 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071779 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.072057 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.071803 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgnk\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-kube-api-access-gvgnk\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.072360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.072333 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.072627 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.072605 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.072755 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.072708 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.072755 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.072744 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.072869 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.072793 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.074674 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.074623 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.075085 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.075063 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.085368 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.085339 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.086447 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.086423 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgnk\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-kube-api-access-gvgnk\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.214836 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.214802 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:37.503666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.503604 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c"] Oct 31 12:54:37.503988 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:54:37.503960 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e07e60_95d3_4a6c_8838_4789a48ff2d6.slice/crio-8ad45b62f5a1d569cce8ac2a1e6a9d59b9892ca6085fb67a81c7d609e5a1ba46 WatchSource:0}: Error finding container 8ad45b62f5a1d569cce8ac2a1e6a9d59b9892ca6085fb67a81c7d609e5a1ba46: Status 404 returned error can't find the container with id 8ad45b62f5a1d569cce8ac2a1e6a9d59b9892ca6085fb67a81c7d609e5a1ba46 Oct 31 12:54:37.505970 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.505925 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:37.506045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.506021 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:37.506083 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:37.506067 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:54:38.510072 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:38.510032 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" event={"ID":"80e07e60-95d3-4a6c-8838-4789a48ff2d6","Type":"ContainerStarted","Data":"745177e4357c15e36b7ce09b6a06b6d472b226a115c88cee66b799e96d436d60"} Oct 31 12:54:38.510489 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:38.510109 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" event={"ID":"80e07e60-95d3-4a6c-8838-4789a48ff2d6","Type":"ContainerStarted","Data":"8ad45b62f5a1d569cce8ac2a1e6a9d59b9892ca6085fb67a81c7d609e5a1ba46"} Oct 31 12:54:38.536465 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:38.536408 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podStartSLOduration=2.536394579 podStartE2EDuration="2.536394579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:38.533792844 +0000 UTC m=+1372.711510827" watchObservedRunningTime="2026-10-31 12:54:38.536394579 +0000 UTC m=+1372.714112562" Oct 31 12:54:39.215608 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:39.215570 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:39.217252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:39.217218 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:54:39.217400 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:39.217285 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 12:54:40.215828 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:40.215786 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:54:40.216302 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:40.215875 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 12:54:41.216041 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:41.216006 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:54:41.216614 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:41.216585 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 12:54:42.216846 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:42.216809 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:54:42.217268 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:42.216874 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 12:54:43.216041 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:43.215997 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 12:54:43.216248 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:43.216064 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 12:54:44.881107 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:44.881072 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:44.883566 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:44.883542 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:44.926939 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:44.926911 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 12:54:46.154138 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.151699 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n"] Oct 31 12:54:46.207697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.199923 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n"] Oct 31 12:54:46.207697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.200082 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.227724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.224305 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"default-dockercfg-5qpt6\"" Oct 31 12:54:46.227724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.224426 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"kube-root-ca.crt\"" Oct 31 12:54:46.227724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.224488 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"openshift-service-ca.crt\"" Oct 31 12:54:46.227724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.224561 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ae9e7\"/\"distinct-w4-7ae9e7-kserve-self-signed-certs\"" Oct 31 12:54:46.460718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.441792 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-model-cache\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.460718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.441862 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tmp-dir\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.460718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.441899 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tls-certs\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.460718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.441925 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-dshm\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.460718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.441989 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-home\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.460718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.442018 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4vd\" (UniqueName: \"kubernetes.io/projected/febb9a1e-921f-4e7d-9af7-a31a8113fbba-kube-api-access-6z4vd\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.549305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.549266 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-home\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.559108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.557908 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4vd\" (UniqueName: \"kubernetes.io/projected/febb9a1e-921f-4e7d-9af7-a31a8113fbba-kube-api-access-6z4vd\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.559108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.558089 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-model-cache\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.559108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.558154 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tmp-dir\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.559108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.558188 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tls-certs\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.559108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.558212 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-dshm\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.565267 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.564123 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-model-cache\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.565267 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.550186 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-home\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.565267 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.564763 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tmp-dir\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.568189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.568143 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-dshm\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.586486 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.586423 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4vd\" (UniqueName: \"kubernetes.io/projected/febb9a1e-921f-4e7d-9af7-a31a8113fbba-kube-api-access-6z4vd\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.616345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.609259 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tls-certs\") pod \"distinct-w4-7ae9e7-kserve-869bd94887-sgq9n\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:46.887664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:46.887598 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:47.442370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:47.421374 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n"] Oct 31 12:54:47.513515 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:54:47.434123 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfebb9a1e_921f_4e7d_9af7_a31a8113fbba.slice/crio-9d5dc7615cb730770721c51c355888214a6cbc8a66965c82a2a7a35bb7c899d1 WatchSource:0}: Error finding container 9d5dc7615cb730770721c51c355888214a6cbc8a66965c82a2a7a35bb7c899d1: Status 404 returned error can't find the container with id 9d5dc7615cb730770721c51c355888214a6cbc8a66965c82a2a7a35bb7c899d1 Oct 31 12:54:47.846320 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:47.846278 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" event={"ID":"febb9a1e-921f-4e7d-9af7-a31a8113fbba","Type":"ContainerStarted","Data":"1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2"} Oct 31 12:54:47.846491 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:47.846332 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" event={"ID":"febb9a1e-921f-4e7d-9af7-a31a8113fbba","Type":"ContainerStarted","Data":"9d5dc7615cb730770721c51c355888214a6cbc8a66965c82a2a7a35bb7c899d1"} Oct 31 12:54:47.873695 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:47.873626 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podStartSLOduration=1.8736104249999999 podStartE2EDuration="1.873610425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:47.871316787 +0000 UTC m=+1382.049034772" watchObservedRunningTime="2026-10-31 12:54:47.873610425 +0000 UTC m=+1382.051328409" Oct 31 12:54:47.888678 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:47.888648 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:50.866622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:50.866592 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:54:55.986923 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:54:55.986887 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:54:56.888981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:56.888946 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:56.901787 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:56.901758 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:54:56.932677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:54:56.932648 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:55:02.170498 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.170459 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x"] Oct 31 12:55:02.194657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.179686 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.194657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.193501 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"e2e-conflict-b-2f2da815-kserve-self-signed-certs\"" Oct 31 12:55:02.213362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.213316 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x"] Oct 31 12:55:02.252519 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.252477 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f109bce2-4124-4d64-8d4e-ceab05df347c-tls-certs\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.252724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.252542 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvxd\" (UniqueName: \"kubernetes.io/projected/f109bce2-4124-4d64-8d4e-ceab05df347c-kube-api-access-kpvxd\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.252724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.252648 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-dshm\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.252880 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.252767 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-tmp-dir\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.252880 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.252793 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-home\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.252880 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.252816 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-model-cache\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.353715 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-tmp-dir\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.353773 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-home\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.353804 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-model-cache\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.353883 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f109bce2-4124-4d64-8d4e-ceab05df347c-tls-certs\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.353964 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvxd\" (UniqueName: \"kubernetes.io/projected/f109bce2-4124-4d64-8d4e-ceab05df347c-kube-api-access-kpvxd\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.354012 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-dshm\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.354846 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-model-cache\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.355098 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-tmp-dir\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.356931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.355318 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-home\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.358654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.358615 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-dshm\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.378655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.367202 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f109bce2-4124-4d64-8d4e-ceab05df347c-tls-certs\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.384306 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.384280 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvxd\" (UniqueName: \"kubernetes.io/projected/f109bce2-4124-4d64-8d4e-ceab05df347c-kube-api-access-kpvxd\") pod \"e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.512984 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.512947 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:02.833513 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.833483 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x"] Oct 31 12:55:02.835366 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:55:02.835339 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf109bce2_4124_4d64_8d4e_ceab05df347c.slice/crio-b8175ad79dc3a7faa694868173e8606c25bbdd3c63e24c3cabcdcd5a1557f2e5 WatchSource:0}: Error finding container b8175ad79dc3a7faa694868173e8606c25bbdd3c63e24c3cabcdcd5a1557f2e5: Status 404 returned error can't find the container with id b8175ad79dc3a7faa694868173e8606c25bbdd3c63e24c3cabcdcd5a1557f2e5 Oct 31 12:55:02.972385 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.972349 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" event={"ID":"f109bce2-4124-4d64-8d4e-ceab05df347c","Type":"ContainerStarted","Data":"0c74572d9406e399a5f0afcecdb22bc316740250c51fd93f36eef456853446cc"} Oct 31 12:55:02.972580 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.972395 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" event={"ID":"f109bce2-4124-4d64-8d4e-ceab05df347c","Type":"ContainerStarted","Data":"b8175ad79dc3a7faa694868173e8606c25bbdd3c63e24c3cabcdcd5a1557f2e5"} Oct 31 12:55:02.996008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:02.995950 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" podStartSLOduration=0.995932086 podStartE2EDuration="995.932086ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:02.992083049 +0000 UTC m=+1397.169801051" watchObservedRunningTime="2026-10-31 12:55:02.995932086 +0000 UTC m=+1397.173650071" Oct 31 12:55:03.513901 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:03.513865 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:12.513439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:12.513403 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:12.533040 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:12.533008 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:12.578578 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:12.578543 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:55:13.048824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:13.048789 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:17.815859 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.815824 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp"] Oct 31 12:55:17.820385 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.820363 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.824459 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.824438 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3dd1e5-openshift-default-dockercfg-ktcdl\"" Oct 31 12:55:17.834280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.834256 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp"] Oct 31 12:55:17.957824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.957789 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.957824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.957827 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.958029 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.957934 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.958029 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.958011 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkvx\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-kube-api-access-dvkvx\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.958100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.958058 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.958100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.958087 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.958173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.958111 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.958173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.958128 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:17.958276 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:17.958209 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059097 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059062 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059425 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059307 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059425 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059426 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059542 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkvx\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-kube-api-access-dvkvx\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059586 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059661 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059707 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059740 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059909 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059788 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.059993 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.059957 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.060151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.060129 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.060425 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.060403 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.060670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.060620 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.060829 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.060731 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.062245 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.062222 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.062939 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.062921 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.072325 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.072265 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.072964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.072920 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkvx\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-kube-api-access-dvkvx\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.137445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.137272 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:18.445845 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:55:18.445814 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3222cd2_22f3_4c4b_8f9b_5e9ef73280eb.slice/crio-f8fef495678e8c1bd3c21a2837902e4e151f8d93f96cdc00b689f0a91c334a88 WatchSource:0}: Error finding container f8fef495678e8c1bd3c21a2837902e4e151f8d93f96cdc00b689f0a91c334a88: Status 404 returned error can't find the container with id f8fef495678e8c1bd3c21a2837902e4e151f8d93f96cdc00b689f0a91c334a88 Oct 31 12:55:18.446193 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.446167 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp"] Oct 31 12:55:18.448433 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.448394 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:18.448520 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.448479 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:18.448563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:18.448526 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:19.199578 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:19.199519 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" event={"ID":"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb","Type":"ContainerStarted","Data":"b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b"} Oct 31 12:55:19.200151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:19.199602 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" event={"ID":"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb","Type":"ContainerStarted","Data":"f8fef495678e8c1bd3c21a2837902e4e151f8d93f96cdc00b689f0a91c334a88"} Oct 31 12:55:19.231917 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:19.231864 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podStartSLOduration=2.231849784 podStartE2EDuration="2.231849784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:19.227337692 +0000 UTC m=+1413.405055693" watchObservedRunningTime="2026-10-31 12:55:19.231849784 +0000 UTC m=+1413.409567767" Oct 31 12:55:20.137750 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:20.137704 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:20.139139 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:20.139107 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:55:20.139266 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:20.139190 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 12:55:21.138219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:21.138171 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:55:21.138758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:21.138233 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 12:55:22.138175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:22.138128 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:55:22.138381 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:22.138199 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 12:55:23.138362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:23.138327 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:55:23.138545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:23.138396 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 12:55:24.137933 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:24.137891 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 12:55:24.138182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:24.137970 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 12:55:25.498716 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:25.498678 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:25.501307 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:25.500051 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:25.501739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:25.501707 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 12:55:25.983707 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:55:25.983670 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:55:27.252231 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:27.252197 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x"] Oct 31 12:55:32.546720 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.546681 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658"] Oct 31 12:55:32.552371 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.552349 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.557380 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.557351 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:55:32.557606 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.557570 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:55:32.557735 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.557709 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:55:32.557902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.557837 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:55:32.560075 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.560041 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658"] Oct 31 12:55:32.640037 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.640002 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.640235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.640072 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72n42\" (UniqueName: \"kubernetes.io/projected/20f8e68d-30f9-46dc-b9e1-3b404add4132-kube-api-access-72n42\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.640235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.640207 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.740980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.740939 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.741187 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.741061 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.741187 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.741091 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72n42\" (UniqueName: \"kubernetes.io/projected/20f8e68d-30f9-46dc-b9e1-3b404add4132-kube-api-access-72n42\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.741830 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.741798 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.743687 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.743664 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.754389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.754364 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72n42\" (UniqueName: \"kubernetes.io/projected/20f8e68d-30f9-46dc-b9e1-3b404add4132-kube-api-access-72n42\") pod \"maas-api-revoke-sub-891c6a76-mf658\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:32.868918 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:32.868802 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:33.164055 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:33.163860 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658"] Oct 31 12:55:33.167868 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:55:33.167836 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f8e68d_30f9_46dc_b9e1_3b404add4132.slice/crio-3e1eb96faad55e06312be559c17ca29effcf99d37f859d884320bdd05a63284d WatchSource:0}: Error finding container 3e1eb96faad55e06312be559c17ca29effcf99d37f859d884320bdd05a63284d: Status 404 returned error can't find the container with id 3e1eb96faad55e06312be559c17ca29effcf99d37f859d884320bdd05a63284d Oct 31 12:55:33.328913 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:33.328871 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" event={"ID":"20f8e68d-30f9-46dc-b9e1-3b404add4132","Type":"ContainerStarted","Data":"3e1eb96faad55e06312be559c17ca29effcf99d37f859d884320bdd05a63284d"} Oct 31 12:55:34.338785 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:34.338672 5074 generic.go:356] "Generic (PLEG): container finished" podID="20f8e68d-30f9-46dc-b9e1-3b404add4132" containerID="88c1263411f63fab475af2da7be76ce4b9f48b430626963372dbd715f65cbd49" exitCode=0 Oct 31 12:55:34.339201 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:34.338769 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" event={"ID":"20f8e68d-30f9-46dc-b9e1-3b404add4132","Type":"ContainerDied","Data":"88c1263411f63fab475af2da7be76ce4b9f48b430626963372dbd715f65cbd49"} Oct 31 12:55:35.516834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.516801 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:35.571286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.571248 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-xks-ca\") pod \"20f8e68d-30f9-46dc-b9e1-3b404add4132\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " Oct 31 12:55:35.571618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.571594 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f8e68d-30f9-46dc-b9e1-3b404add4132-kube-api-access-72n42\" (UniqueName: \"kubernetes.io/projected/20f8e68d-30f9-46dc-b9e1-3b404add4132-kube-api-access-72n42\") pod \"20f8e68d-30f9-46dc-b9e1-3b404add4132\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " Oct 31 12:55:35.571965 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.571943 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-openshift-ca\") pod \"20f8e68d-30f9-46dc-b9e1-3b404add4132\" (UID: \"20f8e68d-30f9-46dc-b9e1-3b404add4132\") " Oct 31 12:55:35.572345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.572319 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-openshift-ca" pod "20f8e68d-30f9-46dc-b9e1-3b404add4132" (UID: "20f8e68d-30f9-46dc-b9e1-3b404add4132"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:35.572740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.572717 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:35.574020 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.573995 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-xks-ca" pod "20f8e68d-30f9-46dc-b9e1-3b404add4132" (UID: "20f8e68d-30f9-46dc-b9e1-3b404add4132"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:35.574464 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.574435 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f8e68d-30f9-46dc-b9e1-3b404add4132-kube-api-access-72n42" pod "20f8e68d-30f9-46dc-b9e1-3b404add4132" (UID: "20f8e68d-30f9-46dc-b9e1-3b404add4132"). InnerVolumeSpecName "kube-api-access-72n42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:35.674516 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.674427 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72n42\" (UniqueName: \"kubernetes.io/projected/20f8e68d-30f9-46dc-b9e1-3b404add4132-kube-api-access-72n42\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:35.674516 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:35.674459 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8e68d-30f9-46dc-b9e1-3b404add4132-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:36.352514 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:36.352478 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" Oct 31 12:55:36.352723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:36.352519 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658" event={"ID":"20f8e68d-30f9-46dc-b9e1-3b404add4132","Type":"ContainerDied","Data":"3e1eb96faad55e06312be559c17ca29effcf99d37f859d884320bdd05a63284d"} Oct 31 12:55:36.352723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:36.352578 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e1eb96faad55e06312be559c17ca29effcf99d37f859d884320bdd05a63284d" Oct 31 12:55:36.996239 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:36.996198 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm"] Oct 31 12:55:37.002153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:36.997147 5074 cpu_manager.go:414] "containerMap: removing container" podUID="20f8e68d-30f9-46dc-b9e1-3b404add4132" containerName="revoke-keys" Oct 31 12:55:37.002153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.000620 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f8e68d-30f9-46dc-b9e1-3b404add4132" containerName="revoke-keys" Oct 31 12:55:37.002153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.000889 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f8e68d-30f9-46dc-b9e1-3b404add4132" containerName="revoke-keys" Oct 31 12:55:37.005170 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.005142 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.012742 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.012700 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm"] Oct 31 12:55:37.095217 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095161 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095217 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095214 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095336 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8zh\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-kube-api-access-kc8zh\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095473 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095507 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73604a4f-c54f-4944-a4b1-6e7519350578-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095548 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095617 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095772 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.095856 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.095838 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73604a4f-c54f-4944-a4b1-6e7519350578-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.196765 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.196731 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.196765 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.196770 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73604a4f-c54f-4944-a4b1-6e7519350578-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197001 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.196794 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197001 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.196922 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197202 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197022 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197202 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197075 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73604a4f-c54f-4944-a4b1-6e7519350578-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197202 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197173 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197202 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197193 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197271 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197200 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197334 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8zh\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-kube-api-access-kc8zh\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197340 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197686 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197602 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73604a4f-c54f-4944-a4b1-6e7519350578-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.197868 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.197719 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.199433 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.199411 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.200044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.200018 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73604a4f-c54f-4944-a4b1-6e7519350578-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.228535 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.228512 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8zh\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-kube-api-access-kc8zh\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.228698 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.228608 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.333020 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.332935 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:37.624162 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.624129 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm"] Oct 31 12:55:37.625494 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:55:37.625464 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73604a4f_c54f_4944_a4b1_6e7519350578.slice/crio-46ee1d5ff55c69b626a60bd865df446cc34f3dc48ce24889ffa8e41f9927bea5 WatchSource:0}: Error finding container 46ee1d5ff55c69b626a60bd865df446cc34f3dc48ce24889ffa8e41f9927bea5: Status 404 returned error can't find the container with id 46ee1d5ff55c69b626a60bd865df446cc34f3dc48ce24889ffa8e41f9927bea5 Oct 31 12:55:37.628100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.628066 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:37.628190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.628134 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:37.628190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:37.628165 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:38.367289 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:38.367251 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" event={"ID":"73604a4f-c54f-4944-a4b1-6e7519350578","Type":"ContainerStarted","Data":"3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa"} Oct 31 12:55:38.367745 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:38.367310 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" event={"ID":"73604a4f-c54f-4944-a4b1-6e7519350578","Type":"ContainerStarted","Data":"46ee1d5ff55c69b626a60bd865df446cc34f3dc48ce24889ffa8e41f9927bea5"} Oct 31 12:55:38.413662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:38.413586 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podStartSLOduration=2.413567381 podStartE2EDuration="2.413567381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:38.409317224 +0000 UTC m=+1432.587035208" watchObservedRunningTime="2026-10-31 12:55:38.413567381 +0000 UTC m=+1432.591285365" Oct 31 12:55:38.966087 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:38.966049 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg"] Oct 31 12:55:38.974361 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:38.974331 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:38.994664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:38.993927 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"distinct-w3-47bcd9-kserve-self-signed-certs\"" Oct 31 12:55:39.003659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.000724 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg"] Oct 31 12:55:39.132150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.132106 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-dshm\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.132467 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.132445 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-home\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.132716 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.132693 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-tmp-dir\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.132865 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.132849 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjfr\" (UniqueName: \"kubernetes.io/projected/296a60e8-1efa-44da-bed6-842ccefdd93d-kube-api-access-8kjfr\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.133049 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.133034 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-model-cache\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.133156 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.133142 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a60e8-1efa-44da-bed6-842ccefdd93d-tls-certs\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.239800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.239682 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-dshm\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.239800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.239769 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-home\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.240049 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.239880 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-tmp-dir\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.240049 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.239930 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjfr\" (UniqueName: \"kubernetes.io/projected/296a60e8-1efa-44da-bed6-842ccefdd93d-kube-api-access-8kjfr\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.240049 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.240013 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-model-cache\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.240278 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.240057 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a60e8-1efa-44da-bed6-842ccefdd93d-tls-certs\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.240772 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.240415 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-home\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.241003 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.240978 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-tmp-dir\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.241368 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.241348 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-model-cache\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.244186 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.243933 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a60e8-1efa-44da-bed6-842ccefdd93d-tls-certs\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.244186 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.244124 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-dshm\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.254273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.254229 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjfr\" (UniqueName: \"kubernetes.io/projected/296a60e8-1efa-44da-bed6-842ccefdd93d-kube-api-access-8kjfr\") pod \"distinct-w3-47bcd9-kserve-65c66584c5-dbqwg\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.334052 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.334011 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:39.335119 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.335100 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:39.335543 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.335523 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:39.335613 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.335566 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:39.705710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.694814 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg"] Oct 31 12:55:39.733390 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:39.733364 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:55:40.333987 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:40.333942 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:40.334203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:40.334009 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:40.411642 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:40.411596 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" event={"ID":"296a60e8-1efa-44da-bed6-842ccefdd93d","Type":"ContainerStarted","Data":"d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2"} Oct 31 12:55:40.411851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:40.411666 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" event={"ID":"296a60e8-1efa-44da-bed6-842ccefdd93d","Type":"ContainerStarted","Data":"985a03abd44218fac271b9980c5b88dfa32b7938db53781ba8f02776c4d938d5"} Oct 31 12:55:40.434922 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:40.434862 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podStartSLOduration=2.434849304 podStartE2EDuration="2.434849304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:40.432226305 +0000 UTC m=+1434.609944290" watchObservedRunningTime="2026-10-31 12:55:40.434849304 +0000 UTC m=+1434.612567288" Oct 31 12:55:41.335155 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:41.335115 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:41.335877 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:41.335758 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:41.337238 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:41.335829 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:42.269780 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:42.269725 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:42.309907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:42.309815 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerName="main" containerID="cri-o://0c74572d9406e399a5f0afcecdb22bc316740250c51fd93f36eef456853446cc" gracePeriod=45 Oct 31 12:55:42.333662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:42.333342 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:42.333662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:42.333406 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:42.457721 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:42.457338 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": EOF" Oct 31 12:55:43.039281 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.039225 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused" Oct 31 12:55:43.039662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.039611 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:43.339711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.338087 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:43.339711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.338157 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:43.547891 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.547858 5074 generic.go:356] "Generic (PLEG): container finished" podID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerID="0c74572d9406e399a5f0afcecdb22bc316740250c51fd93f36eef456853446cc" exitCode=0 Oct 31 12:55:43.548703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.548679 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" event={"ID":"f109bce2-4124-4d64-8d4e-ceab05df347c","Type":"ContainerDied","Data":"0c74572d9406e399a5f0afcecdb22bc316740250c51fd93f36eef456853446cc"} Oct 31 12:55:43.615837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.615725 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.648755 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.648708 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.742217 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.739798 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr"] Oct 31 12:55:43.757193 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.748873 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.868708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.868538 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr"] Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869204 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869273 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869300 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869390 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfzv\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-kube-api-access-vlfzv\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869445 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869543 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869567 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869624 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.872654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.869719 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.964088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.964044 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998280 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998331 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998376 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998438 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998470 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998508 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998528 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998606 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfzv\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-kube-api-access-vlfzv\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:43.998664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.998672 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.002711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.999357 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-credential-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.002711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.999601 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-data\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.002711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:43.999848 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-socket\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.002711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.000324 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-certs\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.015605 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.015562 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istiod-ca-cert\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.082393 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.080985 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-token\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.142476 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.142315 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-envoy\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.167865 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.148997 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfzv\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-kube-api-access-vlfzv\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.167865 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.151060 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-podinfo\") pod \"e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.194686 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.174851 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:44.433714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.433260 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:44.433714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.433323 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:44.907320 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:44.885824 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:45.014706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.009905 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-model-cache\") pod \"f109bce2-4124-4d64-8d4e-ceab05df347c\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " Oct 31 12:55:45.014706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.010091 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f109bce2-4124-4d64-8d4e-ceab05df347c-kube-api-access-kpvxd\" (UniqueName: \"kubernetes.io/projected/f109bce2-4124-4d64-8d4e-ceab05df347c-kube-api-access-kpvxd\") pod \"f109bce2-4124-4d64-8d4e-ceab05df347c\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " Oct 31 12:55:45.014706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.010173 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-home\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-home\") pod \"f109bce2-4124-4d64-8d4e-ceab05df347c\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " Oct 31 12:55:45.014706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.010237 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f109bce2-4124-4d64-8d4e-ceab05df347c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f109bce2-4124-4d64-8d4e-ceab05df347c-tls-certs\") pod \"f109bce2-4124-4d64-8d4e-ceab05df347c\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " Oct 31 12:55:45.014706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.010267 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-tmp-dir\") pod \"f109bce2-4124-4d64-8d4e-ceab05df347c\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " Oct 31 12:55:45.014706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.010314 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-dshm\") pod \"f109bce2-4124-4d64-8d4e-ceab05df347c\" (UID: \"f109bce2-4124-4d64-8d4e-ceab05df347c\") " Oct 31 12:55:45.015456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.015422 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-home" pod "f109bce2-4124-4d64-8d4e-ceab05df347c" (UID: "f109bce2-4124-4d64-8d4e-ceab05df347c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:45.015820 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.015799 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-model-cache" pod "f109bce2-4124-4d64-8d4e-ceab05df347c" (UID: "f109bce2-4124-4d64-8d4e-ceab05df347c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:45.040708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.037729 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-tmp-dir" pod "f109bce2-4124-4d64-8d4e-ceab05df347c" (UID: "f109bce2-4124-4d64-8d4e-ceab05df347c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:45.054311 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.054269 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f109bce2-4124-4d64-8d4e-ceab05df347c-tls-certs" pod "f109bce2-4124-4d64-8d4e-ceab05df347c" (UID: "f109bce2-4124-4d64-8d4e-ceab05df347c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:45.079370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.079328 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:45.121068 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.120296 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:45.121068 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.120336 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f109bce2-4124-4d64-8d4e-ceab05df347c-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:45.121068 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.120350 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:45.121068 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.120366 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:45.145024 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.143729 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:45.145024 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.143824 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:45.145024 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.143871 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:55:45.208667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.200385 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr"] Oct 31 12:55:45.215844 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.215755 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-dshm" pod "f109bce2-4124-4d64-8d4e-ceab05df347c" (UID: "f109bce2-4124-4d64-8d4e-ceab05df347c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:55:45.221825 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.221679 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f109bce2-4124-4d64-8d4e-ceab05df347c-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:45.225860 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.225823 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f109bce2-4124-4d64-8d4e-ceab05df347c-kube-api-access-kpvxd" pod "f109bce2-4124-4d64-8d4e-ceab05df347c" (UID: "f109bce2-4124-4d64-8d4e-ceab05df347c"). InnerVolumeSpecName "kube-api-access-kpvxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:45.328656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.327507 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvxd\" (UniqueName: \"kubernetes.io/projected/f109bce2-4124-4d64-8d4e-ceab05df347c-kube-api-access-kpvxd\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:45.334431 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.334399 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:45.334673 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.334617 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:45.738079 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.738034 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" event={"ID":"f109bce2-4124-4d64-8d4e-ceab05df347c","Type":"ContainerDied","Data":"b8175ad79dc3a7faa694868173e8606c25bbdd3c63e24c3cabcdcd5a1557f2e5"} Oct 31 12:55:45.738360 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.738340 5074 scope.go:122] "RemoveContainer" containerID="0c74572d9406e399a5f0afcecdb22bc316740250c51fd93f36eef456853446cc" Oct 31 12:55:45.738727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.738705 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x" Oct 31 12:55:45.763807 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.763771 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" event={"ID":"7d2f0b9c-1afb-4719-aee9-e73af34dbb26","Type":"ContainerStarted","Data":"4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41"} Oct 31 12:55:45.765740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.765715 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" event={"ID":"7d2f0b9c-1afb-4719-aee9-e73af34dbb26","Type":"ContainerStarted","Data":"1496f533ae26cb38eb55bb3f238232371ef33c9b2bdd5de13500d017564fc3b9"} Oct 31 12:55:45.806381 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.803715 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podStartSLOduration=2.803696611 podStartE2EDuration="2.803696611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:45.792080346 +0000 UTC m=+1439.969798331" watchObservedRunningTime="2026-10-31 12:55:45.803696611 +0000 UTC m=+1439.981414613" Oct 31 12:55:45.817286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.817248 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x"] Oct 31 12:55:45.822345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.822316 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/e2e-conflict-b-2f2da815-kserve-57b59d6b74-xfr7x"] Oct 31 12:55:45.944088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:45.944051 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" path="/var/lib/kubelet/pods/f109bce2-4124-4d64-8d4e-ceab05df347c/volumes" Oct 31 12:55:46.180543 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:46.180395 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:46.182673 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:46.182614 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:55:46.183251 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:46.182987 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 12:55:46.334137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:46.334056 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 12:55:46.334137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:46.334123 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 12:55:47.185511 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.185039 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:55:47.185511 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.185110 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 12:55:47.817538 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.817470 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95"] Oct 31 12:55:47.818414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.818383 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerName="main" Oct 31 12:55:47.818414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.818414 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerName="main" Oct 31 12:55:47.818744 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.818595 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f109bce2-4124-4d64-8d4e-ceab05df347c" containerName="main" Oct 31 12:55:47.832706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.832675 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95"] Oct 31 12:55:47.832857 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.832805 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.843479 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.843450 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbbz\" (UniqueName: \"kubernetes.io/projected/1d38372e-8ddb-4468-9c91-872546af224c-kube-api-access-9hbbz\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.843648 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.843563 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.843733 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.843682 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.844928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.844906 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:55:47.845087 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.844991 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:55:47.845087 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.845075 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:55:47.845265 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.845117 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:55:47.944683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.944340 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbbz\" (UniqueName: \"kubernetes.io/projected/1d38372e-8ddb-4468-9c91-872546af224c-kube-api-access-9hbbz\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.944683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.944426 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.944683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.944523 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.945351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.945318 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.950292 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.950264 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:47.961370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:47.961343 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbbz\" (UniqueName: \"kubernetes.io/projected/1d38372e-8ddb-4468-9c91-872546af224c-kube-api-access-9hbbz\") pod \"maas-api-revoke-sub-1bc36eff-w7z95\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:48.161263 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.161178 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:48.183657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.182036 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:55:48.183657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.182095 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 12:55:48.395657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.393793 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:48.395657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.393860 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:48.542252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.542123 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95"] Oct 31 12:55:48.546108 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:55:48.546072 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d38372e_8ddb_4468_9c91_872546af224c.slice/crio-1b5c324f301d015ce8bb97f8c181eb161a2ea5dbdc9a3c7cf39f09d3227a36df WatchSource:0}: Error finding container 1b5c324f301d015ce8bb97f8c181eb161a2ea5dbdc9a3c7cf39f09d3227a36df: Status 404 returned error can't find the container with id 1b5c324f301d015ce8bb97f8c181eb161a2ea5dbdc9a3c7cf39f09d3227a36df Oct 31 12:55:48.748531 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.748465 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq"] Oct 31 12:55:48.754426 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.754405 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.763517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.763489 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq"] Oct 31 12:55:48.788656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.782955 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:48.796700 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.794095 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:48.808660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.806115 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 12:55:48.825609 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.825570 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" event={"ID":"1d38372e-8ddb-4468-9c91-872546af224c","Type":"ContainerStarted","Data":"1b5c324f301d015ce8bb97f8c181eb161a2ea5dbdc9a3c7cf39f09d3227a36df"} Oct 31 12:55:48.858044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.858012 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.858316 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.858299 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdtv5\" (UniqueName: \"kubernetes.io/projected/58a88d7a-ce6b-49aa-9be2-014e0df588c6-kube-api-access-vdtv5\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.858593 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.858574 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.961673 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.961609 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.961858 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.961700 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdtv5\" (UniqueName: \"kubernetes.io/projected/58a88d7a-ce6b-49aa-9be2-014e0df588c6-kube-api-access-vdtv5\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.961858 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.961793 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.962502 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.962438 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:48.965170 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:48.965146 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:49.021337 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.021303 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdtv5\" (UniqueName: \"kubernetes.io/projected/58a88d7a-ce6b-49aa-9be2-014e0df588c6-kube-api-access-vdtv5\") pod \"maas-api-revoke-sub-1739a7d6-w8blq\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:49.098205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.097843 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:49.182655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.180982 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:55:49.182655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.181042 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 12:55:49.335615 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.335584 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:49.348364 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.348293 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:49.408703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.407901 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq"] Oct 31 12:55:49.833519 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.833487 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" event={"ID":"58a88d7a-ce6b-49aa-9be2-014e0df588c6","Type":"ContainerStarted","Data":"e3bf0776ad21ae6e0626a31e6558e2ff00f37274067a5d917fdef531cef00238"} Oct 31 12:55:49.833690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.833542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" event={"ID":"58a88d7a-ce6b-49aa-9be2-014e0df588c6","Type":"ContainerStarted","Data":"bda9cc4729d15c00c9af84792ef713ef7282551fab5eaa083d7eec3233dd0aa7"} Oct 31 12:55:49.835323 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.835301 5074 generic.go:356] "Generic (PLEG): container finished" podID="1d38372e-8ddb-4468-9c91-872546af224c" containerID="b74878cfe39219c6822af49e33f6bcdeb7d519ca5e06477aba823d304b292ff7" exitCode=0 Oct 31 12:55:49.835449 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.835373 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" event={"ID":"1d38372e-8ddb-4468-9c91-872546af224c","Type":"ContainerDied","Data":"b74878cfe39219c6822af49e33f6bcdeb7d519ca5e06477aba823d304b292ff7"} Oct 31 12:55:49.847412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.847388 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:55:49.852455 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:49.852403 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" podStartSLOduration=1.852390342 podStartE2EDuration="1.852390342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:49.85183519 +0000 UTC m=+1444.029553177" watchObservedRunningTime="2026-10-31 12:55:49.852390342 +0000 UTC m=+1444.030108327" Oct 31 12:55:50.183875 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:50.183836 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:55:50.184155 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:50.184124 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 12:55:50.845091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:50.844992 5074 generic.go:356] "Generic (PLEG): container finished" podID="58a88d7a-ce6b-49aa-9be2-014e0df588c6" containerID="e3bf0776ad21ae6e0626a31e6558e2ff00f37274067a5d917fdef531cef00238" exitCode=0 Oct 31 12:55:50.845091 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:50.845077 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" event={"ID":"58a88d7a-ce6b-49aa-9be2-014e0df588c6","Type":"ContainerDied","Data":"e3bf0776ad21ae6e0626a31e6558e2ff00f37274067a5d917fdef531cef00238"} Oct 31 12:55:51.129064 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.128990 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:51.192078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.192034 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:55:51.192278 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.192102 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 12:55:51.224353 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.224308 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-xks-ca\") pod \"1d38372e-8ddb-4468-9c91-872546af224c\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " Oct 31 12:55:51.224574 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.224408 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d38372e-8ddb-4468-9c91-872546af224c-kube-api-access-9hbbz\" (UniqueName: \"kubernetes.io/projected/1d38372e-8ddb-4468-9c91-872546af224c-kube-api-access-9hbbz\") pod \"1d38372e-8ddb-4468-9c91-872546af224c\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " Oct 31 12:55:51.224574 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.224431 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-openshift-ca\") pod \"1d38372e-8ddb-4468-9c91-872546af224c\" (UID: \"1d38372e-8ddb-4468-9c91-872546af224c\") " Oct 31 12:55:51.225012 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.224987 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-openshift-ca" pod "1d38372e-8ddb-4468-9c91-872546af224c" (UID: "1d38372e-8ddb-4468-9c91-872546af224c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:51.227280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.227246 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d38372e-8ddb-4468-9c91-872546af224c-kube-api-access-9hbbz" pod "1d38372e-8ddb-4468-9c91-872546af224c" (UID: "1d38372e-8ddb-4468-9c91-872546af224c"). InnerVolumeSpecName "kube-api-access-9hbbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:51.228075 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.228052 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-xks-ca" pod "1d38372e-8ddb-4468-9c91-872546af224c" (UID: "1d38372e-8ddb-4468-9c91-872546af224c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:51.325382 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.325345 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:51.325382 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.325379 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d38372e-8ddb-4468-9c91-872546af224c-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:51.325382 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.325393 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbbz\" (UniqueName: \"kubernetes.io/projected/1d38372e-8ddb-4468-9c91-872546af224c-kube-api-access-9hbbz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:51.858405 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.858357 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" Oct 31 12:55:51.858895 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.858408 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95" event={"ID":"1d38372e-8ddb-4468-9c91-872546af224c","Type":"ContainerDied","Data":"1b5c324f301d015ce8bb97f8c181eb161a2ea5dbdc9a3c7cf39f09d3227a36df"} Oct 31 12:55:51.858895 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:51.858530 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b5c324f301d015ce8bb97f8c181eb161a2ea5dbdc9a3c7cf39f09d3227a36df" Oct 31 12:55:52.032781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.032756 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:52.137929 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.137845 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a88d7a-ce6b-49aa-9be2-014e0df588c6-kube-api-access-vdtv5\" (UniqueName: \"kubernetes.io/projected/58a88d7a-ce6b-49aa-9be2-014e0df588c6-kube-api-access-vdtv5\") pod \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " Oct 31 12:55:52.138098 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.137930 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-xks-ca\") pod \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " Oct 31 12:55:52.138098 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.138019 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-openshift-ca\") pod \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\" (UID: \"58a88d7a-ce6b-49aa-9be2-014e0df588c6\") " Oct 31 12:55:52.138415 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.138389 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-openshift-ca" pod "58a88d7a-ce6b-49aa-9be2-014e0df588c6" (UID: "58a88d7a-ce6b-49aa-9be2-014e0df588c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:55:52.138544 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.138527 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:52.140039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.140011 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-xks-ca" pod "58a88d7a-ce6b-49aa-9be2-014e0df588c6" (UID: "58a88d7a-ce6b-49aa-9be2-014e0df588c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:55:52.140129 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.140064 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a88d7a-ce6b-49aa-9be2-014e0df588c6-kube-api-access-vdtv5" pod "58a88d7a-ce6b-49aa-9be2-014e0df588c6" (UID: "58a88d7a-ce6b-49aa-9be2-014e0df588c6"). InnerVolumeSpecName "kube-api-access-vdtv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:55:52.179427 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.179386 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 12:55:52.179550 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.179465 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 12:55:52.239651 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.239594 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdtv5\" (UniqueName: \"kubernetes.io/projected/58a88d7a-ce6b-49aa-9be2-014e0df588c6-kube-api-access-vdtv5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:52.239651 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.239660 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a88d7a-ce6b-49aa-9be2-014e0df588c6-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:55:52.866061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.866032 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" Oct 31 12:55:52.866061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.866045 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq" event={"ID":"58a88d7a-ce6b-49aa-9be2-014e0df588c6","Type":"ContainerDied","Data":"bda9cc4729d15c00c9af84792ef713ef7282551fab5eaa083d7eec3233dd0aa7"} Oct 31 12:55:52.866489 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:52.866094 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda9cc4729d15c00c9af84792ef713ef7282551fab5eaa083d7eec3233dd0aa7" Oct 31 12:55:53.184459 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:53.184365 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:53.184764 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:53.184744 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:53.185380 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:53.185356 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 12:55:56.136134 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:55:56.134070 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:55:56.545659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:56.543745 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:55:56.545659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:56.543819 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:55:58.485777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.485698 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862"] Oct 31 12:55:58.525087 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.525046 5074 cpu_manager.go:414] "containerMap: removing container" podUID="58a88d7a-ce6b-49aa-9be2-014e0df588c6" containerName="revoke-keys" Oct 31 12:55:58.525292 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.525274 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a88d7a-ce6b-49aa-9be2-014e0df588c6" containerName="revoke-keys" Oct 31 12:55:58.525399 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.525386 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1d38372e-8ddb-4468-9c91-872546af224c" containerName="revoke-keys" Oct 31 12:55:58.525481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.525469 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d38372e-8ddb-4468-9c91-872546af224c" containerName="revoke-keys" Oct 31 12:55:58.525867 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.525844 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a88d7a-ce6b-49aa-9be2-014e0df588c6" containerName="revoke-keys" Oct 31 12:55:58.526011 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.525998 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d38372e-8ddb-4468-9c91-872546af224c" containerName="revoke-keys" Oct 31 12:55:58.530362 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.530339 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862"] Oct 31 12:55:58.530588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.530543 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.549505 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.549456 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-47bcd9\"/\"distinct-2-w3-47bcd9-kserve-self-signed-certs\"" Oct 31 12:55:58.656378 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.656322 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c03650-44c5-4c06-b8a3-940deaa36257-tls-certs\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.656622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.656600 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plp4f\" (UniqueName: \"kubernetes.io/projected/66c03650-44c5-4c06-b8a3-940deaa36257-kube-api-access-plp4f\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.656876 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.656852 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-dshm\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.657703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.657679 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-model-cache\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.657956 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.657941 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-home\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.658074 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.658061 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-tmp-dir\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.761727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.761600 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-home\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.762056 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.762020 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-tmp-dir\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.762343 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.762322 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c03650-44c5-4c06-b8a3-940deaa36257-tls-certs\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.762531 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.762509 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plp4f\" (UniqueName: \"kubernetes.io/projected/66c03650-44c5-4c06-b8a3-940deaa36257-kube-api-access-plp4f\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.762731 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.762715 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-dshm\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.762897 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.762869 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-model-cache\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.763548 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.763523 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-model-cache\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.764805 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.764783 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-home\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.765218 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.765199 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-tmp-dir\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.791680 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.787312 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c03650-44c5-4c06-b8a3-940deaa36257-tls-certs\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.800225 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.791936 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-dshm\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.815507 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.815473 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plp4f\" (UniqueName: \"kubernetes.io/projected/66c03650-44c5-4c06-b8a3-940deaa36257-kube-api-access-plp4f\") pod \"distinct-2-w3-47bcd9-kserve-6df5484588-dm862\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.896120 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.896083 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:55:58.997491 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:58.992133 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd"] Oct 31 12:55:59.000795 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.000762 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.020980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.007915 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd"] Oct 31 12:55:59.020980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.008225 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:55:59.020980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.008881 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:55:59.020980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.008997 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:55:59.020980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.009080 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:55:59.067290 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.067257 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.067461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.067304 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75smz\" (UniqueName: \"kubernetes.io/projected/c5896bbb-8913-4ab9-9c61-563f7f416473-kube-api-access-75smz\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.067461 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.067349 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.113288 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.113245 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r"] Oct 31 12:55:59.117273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.117251 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.126956 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.126930 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r"] Oct 31 12:55:59.168441 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.168404 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gt6q\" (UniqueName: \"kubernetes.io/projected/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-kube-api-access-5gt6q\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.168609 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.168501 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.168609 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.168545 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.168609 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.168567 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75smz\" (UniqueName: \"kubernetes.io/projected/c5896bbb-8913-4ab9-9c61-563f7f416473-kube-api-access-75smz\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.168818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.168692 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.168818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.168731 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.169299 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.169274 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.172887 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.172862 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.180915 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.180891 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75smz\" (UniqueName: \"kubernetes.io/projected/c5896bbb-8913-4ab9-9c61-563f7f416473-kube-api-access-75smz\") pod \"maas-api-revoke-sub-08ccdd27-xk7zd\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.193300 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.193179 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862"] Oct 31 12:55:59.269762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.269518 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gt6q\" (UniqueName: \"kubernetes.io/projected/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-kube-api-access-5gt6q\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.269762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.269604 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.269762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.269713 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.270377 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.270349 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.277701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.272589 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.284453 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.284425 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gt6q\" (UniqueName: \"kubernetes.io/projected/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-kube-api-access-5gt6q\") pod \"maas-api-revoke-sub-9ee10d77-vkt5r\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.329835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.329795 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:55:59.433735 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.433705 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:55:59.626881 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:55:59.626821 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5896bbb_8913_4ab9_9c61_563f7f416473.slice/crio-fe4cd1ec7fa2a93ca7212929436833f10a211c41140f3ba95f09e679d135bbca WatchSource:0}: Error finding container fe4cd1ec7fa2a93ca7212929436833f10a211c41140f3ba95f09e679d135bbca: Status 404 returned error can't find the container with id fe4cd1ec7fa2a93ca7212929436833f10a211c41140f3ba95f09e679d135bbca Oct 31 12:55:59.628364 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.628256 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd"] Oct 31 12:55:59.775418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:55:59.775375 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r"] Oct 31 12:55:59.775837 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:55:59.775795 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d9a82f_53c8_47b0_9f31_bed85e3edf8f.slice/crio-b80026a7db812ae5cf396b4421b144d98d36b72802cefde4639698055a8b21e3 WatchSource:0}: Error finding container b80026a7db812ae5cf396b4421b144d98d36b72802cefde4639698055a8b21e3: Status 404 returned error can't find the container with id b80026a7db812ae5cf396b4421b144d98d36b72802cefde4639698055a8b21e3 Oct 31 12:56:00.104345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.104277 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" event={"ID":"66c03650-44c5-4c06-b8a3-940deaa36257","Type":"ContainerStarted","Data":"b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b"} Oct 31 12:56:00.104345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.104323 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" event={"ID":"66c03650-44c5-4c06-b8a3-940deaa36257","Type":"ContainerStarted","Data":"a3aa34e4d4348a13402f5d9ece1b93a8b132ecc62cdca4e10d29c9798c3644e5"} Oct 31 12:56:00.106042 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.106000 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" event={"ID":"11d9a82f-53c8-47b0-9f31-bed85e3edf8f","Type":"ContainerStarted","Data":"b80026a7db812ae5cf396b4421b144d98d36b72802cefde4639698055a8b21e3"} Oct 31 12:56:00.107564 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.107540 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" event={"ID":"c5896bbb-8913-4ab9-9c61-563f7f416473","Type":"ContainerStarted","Data":"ff0a313fd3084db0ddc9c700657eab47a872699981b7536dea6da6a2f796bf87"} Oct 31 12:56:00.107773 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.107579 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" event={"ID":"c5896bbb-8913-4ab9-9c61-563f7f416473","Type":"ContainerStarted","Data":"fe4cd1ec7fa2a93ca7212929436833f10a211c41140f3ba95f09e679d135bbca"} Oct 31 12:56:00.129517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.129428 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podStartSLOduration=2.129413301 podStartE2EDuration="2.129413301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:00.127997878 +0000 UTC m=+1454.305715867" watchObservedRunningTime="2026-10-31 12:56:00.129413301 +0000 UTC m=+1454.307131319" Oct 31 12:56:00.136937 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.136906 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s"] Oct 31 12:56:00.140945 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.140918 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.155575 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.155548 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s"] Oct 31 12:56:00.165588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.165537 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" podStartSLOduration=2.165521013 podStartE2EDuration="2.165521013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:00.151207276 +0000 UTC m=+1454.328925263" watchObservedRunningTime="2026-10-31 12:56:00.165521013 +0000 UTC m=+1454.343238996" Oct 31 12:56:00.184358 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.184322 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlph6\" (UniqueName: \"kubernetes.io/projected/55e1d194-61f7-4bc9-8566-1359db135227-kube-api-access-zlph6\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.184649 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.184609 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.184835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.184811 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.285822 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.285785 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.285972 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.285893 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.288433 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.288387 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.288939 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.288827 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlph6\" (UniqueName: \"kubernetes.io/projected/55e1d194-61f7-4bc9-8566-1359db135227-kube-api-access-zlph6\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.290064 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.290040 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.308025 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.307989 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlph6\" (UniqueName: \"kubernetes.io/projected/55e1d194-61f7-4bc9-8566-1359db135227-kube-api-access-zlph6\") pod \"maas-api-revoke-sub-4c874071-k6l4s\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.456827 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.456747 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:00.749067 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.749040 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s"] Oct 31 12:56:00.758786 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:00.758758 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e1d194_61f7_4bc9_8566_1359db135227.slice/crio-cb9b7aad7bf57266dd7b69a4fbc0f57c39ba9acaeb36bc7ee46ce7c724c0df1c WatchSource:0}: Error finding container cb9b7aad7bf57266dd7b69a4fbc0f57c39ba9acaeb36bc7ee46ce7c724c0df1c: Status 404 returned error can't find the container with id cb9b7aad7bf57266dd7b69a4fbc0f57c39ba9acaeb36bc7ee46ce7c724c0df1c Oct 31 12:56:00.896390 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:00.896358 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:01.115914 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:01.115880 5074 generic.go:356] "Generic (PLEG): container finished" podID="c5896bbb-8913-4ab9-9c61-563f7f416473" containerID="ff0a313fd3084db0ddc9c700657eab47a872699981b7536dea6da6a2f796bf87" exitCode=0 Oct 31 12:56:01.116121 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:01.115970 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" event={"ID":"c5896bbb-8913-4ab9-9c61-563f7f416473","Type":"ContainerDied","Data":"ff0a313fd3084db0ddc9c700657eab47a872699981b7536dea6da6a2f796bf87"} Oct 31 12:56:01.117966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:01.117938 5074 generic.go:356] "Generic (PLEG): container finished" podID="11d9a82f-53c8-47b0-9f31-bed85e3edf8f" containerID="6c6bdebe64b0c787bef244411e33475a58c4ecfb0b137ed7e36ea81e834a92db" exitCode=0 Oct 31 12:56:01.118104 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:01.118015 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" event={"ID":"11d9a82f-53c8-47b0-9f31-bed85e3edf8f","Type":"ContainerDied","Data":"6c6bdebe64b0c787bef244411e33475a58c4ecfb0b137ed7e36ea81e834a92db"} Oct 31 12:56:01.119816 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:01.119789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" event={"ID":"55e1d194-61f7-4bc9-8566-1359db135227","Type":"ContainerStarted","Data":"b27b44a3b0d763a7920026f674b9415601e1cd98628d466733151806f9eecd64"} Oct 31 12:56:01.119921 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:01.119832 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" event={"ID":"55e1d194-61f7-4bc9-8566-1359db135227","Type":"ContainerStarted","Data":"cb9b7aad7bf57266dd7b69a4fbc0f57c39ba9acaeb36bc7ee46ce7c724c0df1c"} Oct 31 12:56:02.129451 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.129411 5074 generic.go:356] "Generic (PLEG): container finished" podID="55e1d194-61f7-4bc9-8566-1359db135227" containerID="b27b44a3b0d763a7920026f674b9415601e1cd98628d466733151806f9eecd64" exitCode=0 Oct 31 12:56:02.129923 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.129514 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" event={"ID":"55e1d194-61f7-4bc9-8566-1359db135227","Type":"ContainerDied","Data":"b27b44a3b0d763a7920026f674b9415601e1cd98628d466733151806f9eecd64"} Oct 31 12:56:02.320108 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.320057 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:56:02.353569 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.353544 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:56:02.414607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.414530 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-xks-ca\") pod \"c5896bbb-8913-4ab9-9c61-563f7f416473\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " Oct 31 12:56:02.414607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.414588 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5896bbb-8913-4ab9-9c61-563f7f416473-kube-api-access-75smz\" (UniqueName: \"kubernetes.io/projected/c5896bbb-8913-4ab9-9c61-563f7f416473-kube-api-access-75smz\") pod \"c5896bbb-8913-4ab9-9c61-563f7f416473\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " Oct 31 12:56:02.414847 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.414672 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-xks-ca\") pod \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " Oct 31 12:56:02.414847 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.414698 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-openshift-ca\") pod \"c5896bbb-8913-4ab9-9c61-563f7f416473\" (UID: \"c5896bbb-8913-4ab9-9c61-563f7f416473\") " Oct 31 12:56:02.414847 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.414745 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-kube-api-access-5gt6q\" (UniqueName: \"kubernetes.io/projected/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-kube-api-access-5gt6q\") pod \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " Oct 31 12:56:02.414987 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.414874 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-openshift-ca\") pod \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\" (UID: \"11d9a82f-53c8-47b0-9f31-bed85e3edf8f\") " Oct 31 12:56:02.415127 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.415100 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-openshift-ca" pod "c5896bbb-8913-4ab9-9c61-563f7f416473" (UID: "c5896bbb-8913-4ab9-9c61-563f7f416473"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:02.415284 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.415255 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-openshift-ca" pod "11d9a82f-53c8-47b0-9f31-bed85e3edf8f" (UID: "11d9a82f-53c8-47b0-9f31-bed85e3edf8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:02.415506 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.415485 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:02.415568 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.415513 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:02.416822 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.416798 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5896bbb-8913-4ab9-9c61-563f7f416473-kube-api-access-75smz" pod "c5896bbb-8913-4ab9-9c61-563f7f416473" (UID: "c5896bbb-8913-4ab9-9c61-563f7f416473"). InnerVolumeSpecName "kube-api-access-75smz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:02.416822 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.416820 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-xks-ca" pod "11d9a82f-53c8-47b0-9f31-bed85e3edf8f" (UID: "11d9a82f-53c8-47b0-9f31-bed85e3edf8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:02.416958 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.416822 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-xks-ca" pod "c5896bbb-8913-4ab9-9c61-563f7f416473" (UID: "c5896bbb-8913-4ab9-9c61-563f7f416473"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:02.416958 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.416835 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-kube-api-access-5gt6q" pod "11d9a82f-53c8-47b0-9f31-bed85e3edf8f" (UID: "11d9a82f-53c8-47b0-9f31-bed85e3edf8f"). InnerVolumeSpecName "kube-api-access-5gt6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:02.516527 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.516495 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:02.516527 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.516525 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gt6q\" (UniqueName: \"kubernetes.io/projected/11d9a82f-53c8-47b0-9f31-bed85e3edf8f-kube-api-access-5gt6q\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:02.516527 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.516536 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5896bbb-8913-4ab9-9c61-563f7f416473-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:02.516817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:02.516549 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75smz\" (UniqueName: \"kubernetes.io/projected/c5896bbb-8913-4ab9-9c61-563f7f416473-kube-api-access-75smz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:03.138445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.138411 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" Oct 31 12:56:03.138445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.138429 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r" event={"ID":"11d9a82f-53c8-47b0-9f31-bed85e3edf8f","Type":"ContainerDied","Data":"b80026a7db812ae5cf396b4421b144d98d36b72802cefde4639698055a8b21e3"} Oct 31 12:56:03.138944 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.138479 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80026a7db812ae5cf396b4421b144d98d36b72802cefde4639698055a8b21e3" Oct 31 12:56:03.144853 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.144828 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" Oct 31 12:56:03.145017 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.144877 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd" event={"ID":"c5896bbb-8913-4ab9-9c61-563f7f416473","Type":"ContainerDied","Data":"fe4cd1ec7fa2a93ca7212929436833f10a211c41140f3ba95f09e679d135bbca"} Oct 31 12:56:03.145017 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.144927 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe4cd1ec7fa2a93ca7212929436833f10a211c41140f3ba95f09e679d135bbca" Oct 31 12:56:03.321110 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.321077 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:03.427406 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.427317 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e1d194-61f7-4bc9-8566-1359db135227-kube-api-access-zlph6\" (UniqueName: \"kubernetes.io/projected/55e1d194-61f7-4bc9-8566-1359db135227-kube-api-access-zlph6\") pod \"55e1d194-61f7-4bc9-8566-1359db135227\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " Oct 31 12:56:03.427579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.427427 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-openshift-ca\") pod \"55e1d194-61f7-4bc9-8566-1359db135227\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " Oct 31 12:56:03.427579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.427466 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-xks-ca\") pod \"55e1d194-61f7-4bc9-8566-1359db135227\" (UID: \"55e1d194-61f7-4bc9-8566-1359db135227\") " Oct 31 12:56:03.427871 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.427836 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-openshift-ca" pod "55e1d194-61f7-4bc9-8566-1359db135227" (UID: "55e1d194-61f7-4bc9-8566-1359db135227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:03.428035 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.428016 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:03.429702 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.429620 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e1d194-61f7-4bc9-8566-1359db135227-kube-api-access-zlph6" pod "55e1d194-61f7-4bc9-8566-1359db135227" (UID: "55e1d194-61f7-4bc9-8566-1359db135227"). InnerVolumeSpecName "kube-api-access-zlph6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:03.429941 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.429912 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-xks-ca" pod "55e1d194-61f7-4bc9-8566-1359db135227" (UID: "55e1d194-61f7-4bc9-8566-1359db135227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:03.529187 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.529148 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlph6\" (UniqueName: \"kubernetes.io/projected/55e1d194-61f7-4bc9-8566-1359db135227-kube-api-access-zlph6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:03.529187 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:03.529180 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e1d194-61f7-4bc9-8566-1359db135227-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:04.152978 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:04.152939 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" event={"ID":"55e1d194-61f7-4bc9-8566-1359db135227","Type":"ContainerDied","Data":"cb9b7aad7bf57266dd7b69a4fbc0f57c39ba9acaeb36bc7ee46ce7c724c0df1c"} Oct 31 12:56:04.152978 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:04.152964 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s" Oct 31 12:56:04.153521 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:04.153003 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb9b7aad7bf57266dd7b69a4fbc0f57c39ba9acaeb36bc7ee46ce7c724c0df1c" Oct 31 12:56:07.027935 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.027901 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w"] Oct 31 12:56:07.028496 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028476 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c5896bbb-8913-4ab9-9c61-563f7f416473" containerName="revoke-keys" Oct 31 12:56:07.028496 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028495 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5896bbb-8913-4ab9-9c61-563f7f416473" containerName="revoke-keys" Oct 31 12:56:07.028710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028510 5074 cpu_manager.go:414] "containerMap: removing container" podUID="11d9a82f-53c8-47b0-9f31-bed85e3edf8f" containerName="revoke-keys" Oct 31 12:56:07.028710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028517 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d9a82f-53c8-47b0-9f31-bed85e3edf8f" containerName="revoke-keys" Oct 31 12:56:07.028710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028534 5074 cpu_manager.go:414] "containerMap: removing container" podUID="55e1d194-61f7-4bc9-8566-1359db135227" containerName="revoke-keys" Oct 31 12:56:07.028710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028542 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e1d194-61f7-4bc9-8566-1359db135227" containerName="revoke-keys" Oct 31 12:56:07.028710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028700 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d9a82f-53c8-47b0-9f31-bed85e3edf8f" containerName="revoke-keys" Oct 31 12:56:07.028943 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028725 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5896bbb-8913-4ab9-9c61-563f7f416473" containerName="revoke-keys" Oct 31 12:56:07.028943 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.028736 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e1d194-61f7-4bc9-8566-1359db135227" containerName="revoke-keys" Oct 31 12:56:07.032564 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.032539 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.054585 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.054562 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w"] Oct 31 12:56:07.170966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.170933 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.170972 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.170994 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171250 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.171178 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171313 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.171291 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171382 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.171368 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171467 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.171446 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171543 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.171527 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.171605 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.171590 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcp7\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-kube-api-access-rtcp7\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.272715 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.272679 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.272957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.272738 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.272957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.272775 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.272957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.272823 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcp7\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-kube-api-access-rtcp7\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.272957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.272926 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.273208 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.272960 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.273208 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.272989 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.273208 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.273051 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.273208 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.273119 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.273526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.273488 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.273699 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.273671 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.274141 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.274114 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.274301 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.274138 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.274406 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.274379 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.275881 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.275858 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.276199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.276165 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.285451 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.285385 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.285546 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.285514 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcp7\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-kube-api-access-rtcp7\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.319229 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.319187 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:56:07.354994 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.354951 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:07.647165 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.647133 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w"] Oct 31 12:56:07.647397 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:07.647370 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ec2bc5_3f35_4fbe_82f8_9351d6017ba4.slice/crio-4d461098cd7ec528a6cc19d8970589cb8f4b6b1fc0616fe420d6e938a6d490fe WatchSource:0}: Error finding container 4d461098cd7ec528a6cc19d8970589cb8f4b6b1fc0616fe420d6e938a6d490fe: Status 404 returned error can't find the container with id 4d461098cd7ec528a6cc19d8970589cb8f4b6b1fc0616fe420d6e938a6d490fe Oct 31 12:56:07.649965 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.649932 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.650098 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.650021 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:07.650098 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:07.650067 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:08.209500 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:08.209461 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" event={"ID":"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4","Type":"ContainerStarted","Data":"39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2"} Oct 31 12:56:08.209928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:08.209530 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" event={"ID":"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4","Type":"ContainerStarted","Data":"4d461098cd7ec528a6cc19d8970589cb8f4b6b1fc0616fe420d6e938a6d490fe"} Oct 31 12:56:08.296747 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:08.296665 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podStartSLOduration=1.296624695 podStartE2EDuration="1.296624695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:08.294145019 +0000 UTC m=+1462.471863005" watchObservedRunningTime="2026-10-31 12:56:08.296624695 +0000 UTC m=+1462.474342679" Oct 31 12:56:08.355654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:08.355560 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:08.364679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:08.357513 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:08.364679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:08.357571 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:08.904659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:08.901594 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:09.358657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:09.358086 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:09.358657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:09.358171 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:09.624728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:09.622741 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:09.936183 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:09.936082 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 12:56:09.943232 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:09.943191 5074 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.359656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.359066 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:10.359656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.359147 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:10.385659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.373716 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.386800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.386751 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.491498 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.491457 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.637717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.625732 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.637717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.626728 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:56:10.861666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.861601 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.954953 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.954838 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:10.975665 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:10.956152 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:56:11.390825 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.390786 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:11.391372 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.391339 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:11.507674 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.495478 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:11.507674 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.496067 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:56:11.620659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.619719 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:11.620659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.619798 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:11.644420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.644291 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:11.777261 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.777224 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:56:11.860744 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.860694 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q"] Oct 31 12:56:11.976888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.940908 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:11.992667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.977415 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:56:11.992667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.978153 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:56:11.992667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.978281 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:56:11.992667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:11.978388 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:56:12.151386 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.130591 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56cmb\" (UniqueName: \"kubernetes.io/projected/6cdb654e-e974-4799-a809-7a51be6baad1-kube-api-access-56cmb\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.151386 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.130776 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.151386 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.130891 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.229721 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.229610 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:56:12.229960 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.229940 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q"] Oct 31 12:56:12.235919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.235488 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.235919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.235611 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.235919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.235715 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56cmb\" (UniqueName: \"kubernetes.io/projected/6cdb654e-e974-4799-a809-7a51be6baad1-kube-api-access-56cmb\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.236434 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.236383 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.241035 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.241012 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.242235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.242218 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:56:12.250430 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.250397 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56cmb\" (UniqueName: \"kubernetes.io/projected/6cdb654e-e974-4799-a809-7a51be6baad1-kube-api-access-56cmb\") pod \"maas-api-revoke-sub-d9511ee0-s4n5q\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.357346 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.357310 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:12.357547 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.357374 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:12.454795 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.454763 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:12.806239 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:12.806200 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q"] Oct 31 12:56:12.808801 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:12.808763 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdb654e_e974_4799_a809_7a51be6baad1.slice/crio-3e44032933ac88cb2105396526001a3f92e78b4b8e965c2957b729437abfadac WatchSource:0}: Error finding container 3e44032933ac88cb2105396526001a3f92e78b4b8e965c2957b729437abfadac: Status 404 returned error can't find the container with id 3e44032933ac88cb2105396526001a3f92e78b4b8e965c2957b729437abfadac Oct 31 12:56:13.355581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.355503 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:13.355581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.355553 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:13.409652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.404280 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566"] Oct 31 12:56:13.414673 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.414603 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.423779 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.423309 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566"] Oct 31 12:56:13.512290 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.512252 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" event={"ID":"6cdb654e-e974-4799-a809-7a51be6baad1","Type":"ContainerStarted","Data":"dd6d633c4d0fd72ba268deb67f61ef2b7702d1f3bd9a24bf67968a7636d3e530"} Oct 31 12:56:13.512942 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.512910 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" event={"ID":"6cdb654e-e974-4799-a809-7a51be6baad1","Type":"ContainerStarted","Data":"3e44032933ac88cb2105396526001a3f92e78b4b8e965c2957b729437abfadac"} Oct 31 12:56:13.535678 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.535602 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" podStartSLOduration=2.535588803 podStartE2EDuration="2.535588803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:13.533854422 +0000 UTC m=+1467.711572407" watchObservedRunningTime="2026-10-31 12:56:13.535588803 +0000 UTC m=+1467.713306786" Oct 31 12:56:13.554612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.554575 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8r2b\" (UniqueName: \"kubernetes.io/projected/f8366c85-e16e-4973-bcc9-35d2f85c5380-kube-api-access-f8r2b\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.554728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.554710 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.554789 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.554745 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.656306 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.656214 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.656471 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.656451 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8r2b\" (UniqueName: \"kubernetes.io/projected/f8366c85-e16e-4973-bcc9-35d2f85c5380-kube-api-access-f8r2b\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.656834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.656535 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.657199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.657176 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.658875 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.658848 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.668889 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.668863 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8r2b\" (UniqueName: \"kubernetes.io/projected/f8366c85-e16e-4973-bcc9-35d2f85c5380-kube-api-access-f8r2b\") pod \"maas-api-revoke-sub-89790774-gd566\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:13.744236 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:13.744201 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:14.108089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.108056 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566"] Oct 31 12:56:14.129305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.129275 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv"] Oct 31 12:56:14.134834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.134799 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.144785 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.144761 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-107955\"/\"distinct-w0-107955-kserve-self-signed-certs\"" Oct 31 12:56:14.148454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.148424 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv"] Oct 31 12:56:14.267046 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.267001 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-model-cache\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.267352 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.267332 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tls-certs\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.267548 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.267533 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-dshm\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.267728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.267711 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjhh\" (UniqueName: \"kubernetes.io/projected/93cbee9f-cd84-4c3b-b818-0ca742df8a11-kube-api-access-vpjhh\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.268022 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.268000 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-home\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.269610 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.269588 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tmp-dir\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.368750 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.368627 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:14.369021 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.368983 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:14.379965 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.379926 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-home\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.380585 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.380480 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-home\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.380852 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.380834 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tmp-dir\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.381297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.381203 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tmp-dir\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.381448 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.381434 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-model-cache\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.381856 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.381840 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tls-certs\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.382415 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.381788 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-model-cache\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.382553 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.382537 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-dshm\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.382980 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.382961 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjhh\" (UniqueName: \"kubernetes.io/projected/93cbee9f-cd84-4c3b-b818-0ca742df8a11-kube-api-access-vpjhh\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.406619 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.405988 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tls-certs\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.409318 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.409276 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-dshm\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.428508 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.425682 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr"] Oct 31 12:56:14.431002 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.430978 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.433949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.433921 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr"] Oct 31 12:56:14.440325 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.436864 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjhh\" (UniqueName: \"kubernetes.io/projected/93cbee9f-cd84-4c3b-b818-0ca742df8a11-kube-api-access-vpjhh\") pod \"distinct-w0-107955-kserve-64bf5b9d-ck2qv\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.463666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.461467 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:14.530572 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.530542 5074 generic.go:356] "Generic (PLEG): container finished" podID="6cdb654e-e974-4799-a809-7a51be6baad1" containerID="dd6d633c4d0fd72ba268deb67f61ef2b7702d1f3bd9a24bf67968a7636d3e530" exitCode=0 Oct 31 12:56:14.531025 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.530720 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" event={"ID":"6cdb654e-e974-4799-a809-7a51be6baad1","Type":"ContainerDied","Data":"dd6d633c4d0fd72ba268deb67f61ef2b7702d1f3bd9a24bf67968a7636d3e530"} Oct 31 12:56:14.532769 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.532702 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" event={"ID":"f8366c85-e16e-4973-bcc9-35d2f85c5380","Type":"ContainerStarted","Data":"bbe2fbbc79313b0989be20af0ca18be65527630ef4c3bb6834149b12f8639daa"} Oct 31 12:56:14.594902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.594478 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.594902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.594659 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6szk\" (UniqueName: \"kubernetes.io/projected/a9ff190b-0b3b-4dfe-be15-127856f947ba-kube-api-access-g6szk\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.594902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.594723 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.702310 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.702268 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6szk\" (UniqueName: \"kubernetes.io/projected/a9ff190b-0b3b-4dfe-be15-127856f947ba-kube-api-access-g6szk\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.702721 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.702698 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.703081 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.703064 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.704070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.704047 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.706134 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.706073 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.717620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.717592 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6szk\" (UniqueName: \"kubernetes.io/projected/a9ff190b-0b3b-4dfe-be15-127856f947ba-kube-api-access-g6szk\") pod \"maas-api-revoke-sub-56318de9-9jmfr\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.817545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.817515 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:14.843341 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:14.843311 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv"] Oct 31 12:56:14.847406 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:14.847380 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93cbee9f_cd84_4c3b_b818_0ca742df8a11.slice/crio-d4b42fe9bfcbcb97bd72ce205fea0dfdc5058e4019adab419108cc8fc632a58c WatchSource:0}: Error finding container d4b42fe9bfcbcb97bd72ce205fea0dfdc5058e4019adab419108cc8fc632a58c: Status 404 returned error can't find the container with id d4b42fe9bfcbcb97bd72ce205fea0dfdc5058e4019adab419108cc8fc632a58c Oct 31 12:56:15.133787 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:15.133756 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ff190b_0b3b_4dfe_be15_127856f947ba.slice/crio-fe1e8ae27a3d6be97c5ffc2e2cb4117cee734b45bafad6cfe76c747d00787647 WatchSource:0}: Error finding container fe1e8ae27a3d6be97c5ffc2e2cb4117cee734b45bafad6cfe76c747d00787647: Status 404 returned error can't find the container with id fe1e8ae27a3d6be97c5ffc2e2cb4117cee734b45bafad6cfe76c747d00787647 Oct 31 12:56:15.134079 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.134054 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr"] Oct 31 12:56:15.356367 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.356324 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 12:56:15.356563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.356389 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 12:56:15.543718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.543616 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" event={"ID":"93cbee9f-cd84-4c3b-b818-0ca742df8a11","Type":"ContainerStarted","Data":"44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9"} Oct 31 12:56:15.543718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.543689 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" event={"ID":"93cbee9f-cd84-4c3b-b818-0ca742df8a11","Type":"ContainerStarted","Data":"d4b42fe9bfcbcb97bd72ce205fea0dfdc5058e4019adab419108cc8fc632a58c"} Oct 31 12:56:15.546313 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.546282 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" event={"ID":"a9ff190b-0b3b-4dfe-be15-127856f947ba","Type":"ContainerStarted","Data":"51825be5a7a3f9e8f0667d17c20b1559fee8d7a2fc6d3e03bd17401035da82ff"} Oct 31 12:56:15.546452 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.546338 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" event={"ID":"a9ff190b-0b3b-4dfe-be15-127856f947ba","Type":"ContainerStarted","Data":"fe1e8ae27a3d6be97c5ffc2e2cb4117cee734b45bafad6cfe76c747d00787647"} Oct 31 12:56:15.548012 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.547984 5074 generic.go:356] "Generic (PLEG): container finished" podID="f8366c85-e16e-4973-bcc9-35d2f85c5380" containerID="d6232fa45375271469796922e72a259422f1af6fe406f9151c15ddf52a9fef45" exitCode=0 Oct 31 12:56:15.548134 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.548073 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" event={"ID":"f8366c85-e16e-4973-bcc9-35d2f85c5380","Type":"ContainerDied","Data":"d6232fa45375271469796922e72a259422f1af6fe406f9151c15ddf52a9fef45"} Oct 31 12:56:15.565096 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.565046 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podStartSLOduration=1.565031495 podStartE2EDuration="1.565031495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:15.562562395 +0000 UTC m=+1469.740280399" watchObservedRunningTime="2026-10-31 12:56:15.565031495 +0000 UTC m=+1469.742749478" Oct 31 12:56:15.602483 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.602426 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" podStartSLOduration=1.602410428 podStartE2EDuration="1.602410428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:15.599401879 +0000 UTC m=+1469.777119863" watchObservedRunningTime="2026-10-31 12:56:15.602410428 +0000 UTC m=+1469.780128411" Oct 31 12:56:15.724743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.724712 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:15.923420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.923334 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-xks-ca\") pod \"6cdb654e-e974-4799-a809-7a51be6baad1\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " Oct 31 12:56:15.923420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.923375 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-openshift-ca\") pod \"6cdb654e-e974-4799-a809-7a51be6baad1\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " Oct 31 12:56:15.923670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.923490 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdb654e-e974-4799-a809-7a51be6baad1-kube-api-access-56cmb\" (UniqueName: \"kubernetes.io/projected/6cdb654e-e974-4799-a809-7a51be6baad1-kube-api-access-56cmb\") pod \"6cdb654e-e974-4799-a809-7a51be6baad1\" (UID: \"6cdb654e-e974-4799-a809-7a51be6baad1\") " Oct 31 12:56:15.923780 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.923756 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-openshift-ca" pod "6cdb654e-e974-4799-a809-7a51be6baad1" (UID: "6cdb654e-e974-4799-a809-7a51be6baad1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:15.924197 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.924170 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:15.925905 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.925849 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-xks-ca" pod "6cdb654e-e974-4799-a809-7a51be6baad1" (UID: "6cdb654e-e974-4799-a809-7a51be6baad1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:15.925905 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:15.925869 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdb654e-e974-4799-a809-7a51be6baad1-kube-api-access-56cmb" pod "6cdb654e-e974-4799-a809-7a51be6baad1" (UID: "6cdb654e-e974-4799-a809-7a51be6baad1"). InnerVolumeSpecName "kube-api-access-56cmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:16.026214 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.026160 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb654e-e974-4799-a809-7a51be6baad1-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:16.026412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.026227 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56cmb\" (UniqueName: \"kubernetes.io/projected/6cdb654e-e974-4799-a809-7a51be6baad1-kube-api-access-56cmb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:16.462846 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.462809 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:16.557964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.557856 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" event={"ID":"6cdb654e-e974-4799-a809-7a51be6baad1","Type":"ContainerDied","Data":"3e44032933ac88cb2105396526001a3f92e78b4b8e965c2957b729437abfadac"} Oct 31 12:56:16.557964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.557931 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e44032933ac88cb2105396526001a3f92e78b4b8e965c2957b729437abfadac" Oct 31 12:56:16.557964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.557877 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q" Oct 31 12:56:16.560170 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.560140 5074 generic.go:356] "Generic (PLEG): container finished" podID="a9ff190b-0b3b-4dfe-be15-127856f947ba" containerID="51825be5a7a3f9e8f0667d17c20b1559fee8d7a2fc6d3e03bd17401035da82ff" exitCode=0 Oct 31 12:56:16.560308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.560283 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" event={"ID":"a9ff190b-0b3b-4dfe-be15-127856f947ba","Type":"ContainerDied","Data":"51825be5a7a3f9e8f0667d17c20b1559fee8d7a2fc6d3e03bd17401035da82ff"} Oct 31 12:56:16.754541 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.754509 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:16.836472 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.836396 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-openshift-ca\") pod \"f8366c85-e16e-4973-bcc9-35d2f85c5380\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " Oct 31 12:56:16.836613 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.836544 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-xks-ca\") pod \"f8366c85-e16e-4973-bcc9-35d2f85c5380\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " Oct 31 12:56:16.836695 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.836655 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8366c85-e16e-4973-bcc9-35d2f85c5380-kube-api-access-f8r2b\" (UniqueName: \"kubernetes.io/projected/f8366c85-e16e-4973-bcc9-35d2f85c5380-kube-api-access-f8r2b\") pod \"f8366c85-e16e-4973-bcc9-35d2f85c5380\" (UID: \"f8366c85-e16e-4973-bcc9-35d2f85c5380\") " Oct 31 12:56:16.836851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.836826 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-openshift-ca" pod "f8366c85-e16e-4973-bcc9-35d2f85c5380" (UID: "f8366c85-e16e-4973-bcc9-35d2f85c5380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:16.837071 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.837048 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:16.838770 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.838738 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-xks-ca" pod "f8366c85-e16e-4973-bcc9-35d2f85c5380" (UID: "f8366c85-e16e-4973-bcc9-35d2f85c5380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:16.838904 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.838804 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8366c85-e16e-4973-bcc9-35d2f85c5380-kube-api-access-f8r2b" pod "f8366c85-e16e-4973-bcc9-35d2f85c5380" (UID: "f8366c85-e16e-4973-bcc9-35d2f85c5380"). InnerVolumeSpecName "kube-api-access-f8r2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:16.937981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.937938 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8r2b\" (UniqueName: \"kubernetes.io/projected/f8366c85-e16e-4973-bcc9-35d2f85c5380-kube-api-access-f8r2b\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:16.937981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:16.937976 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8366c85-e16e-4973-bcc9-35d2f85c5380-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:17.400666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.399760 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:17.400666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.399879 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:17.568218 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.568192 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" Oct 31 12:56:17.568218 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.568216 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566" event={"ID":"f8366c85-e16e-4973-bcc9-35d2f85c5380","Type":"ContainerDied","Data":"bbe2fbbc79313b0989be20af0ca18be65527630ef4c3bb6834149b12f8639daa"} Oct 31 12:56:17.568754 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.568264 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe2fbbc79313b0989be20af0ca18be65527630ef4c3bb6834149b12f8639daa" Oct 31 12:56:17.810928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.810896 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:17.860670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.856721 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-xks-ca\") pod \"a9ff190b-0b3b-4dfe-be15-127856f947ba\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " Oct 31 12:56:17.860670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.856776 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ff190b-0b3b-4dfe-be15-127856f947ba-kube-api-access-g6szk\" (UniqueName: \"kubernetes.io/projected/a9ff190b-0b3b-4dfe-be15-127856f947ba-kube-api-access-g6szk\") pod \"a9ff190b-0b3b-4dfe-be15-127856f947ba\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " Oct 31 12:56:17.860670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.856904 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-openshift-ca\") pod \"a9ff190b-0b3b-4dfe-be15-127856f947ba\" (UID: \"a9ff190b-0b3b-4dfe-be15-127856f947ba\") " Oct 31 12:56:17.860670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.857875 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-openshift-ca" pod "a9ff190b-0b3b-4dfe-be15-127856f947ba" (UID: "a9ff190b-0b3b-4dfe-be15-127856f947ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:17.861617 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.861588 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ff190b-0b3b-4dfe-be15-127856f947ba-kube-api-access-g6szk" pod "a9ff190b-0b3b-4dfe-be15-127856f947ba" (UID: "a9ff190b-0b3b-4dfe-be15-127856f947ba"). InnerVolumeSpecName "kube-api-access-g6szk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:17.865780 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.865749 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-xks-ca" pod "a9ff190b-0b3b-4dfe-be15-127856f947ba" (UID: "a9ff190b-0b3b-4dfe-be15-127856f947ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:17.958989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.958958 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:17.958989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.958989 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ff190b-0b3b-4dfe-be15-127856f947ba-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:17.959173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.959003 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6szk\" (UniqueName: \"kubernetes.io/projected/a9ff190b-0b3b-4dfe-be15-127856f947ba-kube-api-access-g6szk\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:17.983323 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.983295 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:17.983936 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.983853 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:17.984982 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:17.984965 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 12:56:18.008738 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.008704 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7"] Oct 31 12:56:18.009530 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009510 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6cdb654e-e974-4799-a809-7a51be6baad1" containerName="revoke-keys" Oct 31 12:56:18.009592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009534 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdb654e-e974-4799-a809-7a51be6baad1" containerName="revoke-keys" Oct 31 12:56:18.009592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009551 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a9ff190b-0b3b-4dfe-be15-127856f947ba" containerName="revoke-keys" Oct 31 12:56:18.009592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009560 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ff190b-0b3b-4dfe-be15-127856f947ba" containerName="revoke-keys" Oct 31 12:56:18.009592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009580 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f8366c85-e16e-4973-bcc9-35d2f85c5380" containerName="revoke-keys" Oct 31 12:56:18.009592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009588 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8366c85-e16e-4973-bcc9-35d2f85c5380" containerName="revoke-keys" Oct 31 12:56:18.009782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009744 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdb654e-e974-4799-a809-7a51be6baad1" containerName="revoke-keys" Oct 31 12:56:18.009782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009756 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8366c85-e16e-4973-bcc9-35d2f85c5380" containerName="revoke-keys" Oct 31 12:56:18.009782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.009766 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ff190b-0b3b-4dfe-be15-127856f947ba" containerName="revoke-keys" Oct 31 12:56:18.014016 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.013993 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.036600 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.036575 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7"] Oct 31 12:56:18.060595 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.060463 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.060595 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.060581 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.060837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.060698 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.060837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.060734 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.060837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.060803 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.061004 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.060973 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r745d\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-kube-api-access-r745d\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.061060 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.061032 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.061116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.061069 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.061285 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.061244 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.079909 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.079879 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz"] Oct 31 12:56:18.084049 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.084028 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.117932 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.117901 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz"] Oct 31 12:56:18.162323 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.162284 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.162488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.162354 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.162488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.162402 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.162488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.162457 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r745d\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-kube-api-access-r745d\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.162691 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.162492 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdb5206-fc55-415d-bf93-6abad43ae69b-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.162967 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.162936 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.163219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.163006 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.163219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.163035 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.163219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.163103 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.163219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.163137 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.163219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.163175 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.163484 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.163437 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdp6\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-kube-api-access-jsdp6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.164111 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164018 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.164414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164388 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.165152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164585 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.165152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164615 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.165152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164747 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.165152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164776 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.165152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164810 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.165152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164840 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.165152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.164881 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.165623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.165587 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.165859 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.165840 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.167853 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.167826 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.168150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.168127 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.180364 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.180315 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r745d\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-kube-api-access-r745d\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.180959 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.180918 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.266161 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266128 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266201 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266229 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266265 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdb5206-fc55-415d-bf93-6abad43ae69b-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266300 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266314 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266352 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266802 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266387 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdp6\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-kube-api-access-jsdp6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266802 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266416 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266802 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266723 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-credential-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.266802 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.266758 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-certs\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.267045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.267022 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-socket\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.267116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.267097 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-data\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.267412 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.267387 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdb5206-fc55-415d-bf93-6abad43ae69b-istiod-ca-cert\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.269102 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.269079 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-envoy\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.269227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.269206 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-podinfo\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.279116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.279064 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-token\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.279205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.279175 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdp6\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-kube-api-access-jsdp6\") pod \"e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.327554 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.327524 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:18.414959 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.414919 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:18.585282 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.585090 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" event={"ID":"a9ff190b-0b3b-4dfe-be15-127856f947ba","Type":"ContainerDied","Data":"fe1e8ae27a3d6be97c5ffc2e2cb4117cee734b45bafad6cfe76c747d00787647"} Oct 31 12:56:18.585282 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.585194 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe1e8ae27a3d6be97c5ffc2e2cb4117cee734b45bafad6cfe76c747d00787647" Oct 31 12:56:18.585913 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.585892 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr" Oct 31 12:56:18.623050 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.623022 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7"] Oct 31 12:56:18.625284 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:18.625252 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d18089_6846_4b29_bf2d_8b0d1e024b17.slice/crio-06b96559c610dfc272b2cdef7c2785aeced60abc3b33a4c7ac816b5bb38a3f14 WatchSource:0}: Error finding container 06b96559c610dfc272b2cdef7c2785aeced60abc3b33a4c7ac816b5bb38a3f14: Status 404 returned error can't find the container with id 06b96559c610dfc272b2cdef7c2785aeced60abc3b33a4c7ac816b5bb38a3f14 Oct 31 12:56:18.628114 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.628076 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.628212 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.628159 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.628212 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.628201 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.732701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.731344 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz"] Oct 31 12:56:18.732701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.732165 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.732701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.732318 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.732701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.732450 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:56:18.917462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.917430 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:18.934702 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:18.934670 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:19.626656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:19.624673 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" event={"ID":"9bdb5206-fc55-415d-bf93-6abad43ae69b","Type":"ContainerStarted","Data":"f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868"} Oct 31 12:56:19.626656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:19.624747 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" event={"ID":"9bdb5206-fc55-415d-bf93-6abad43ae69b","Type":"ContainerStarted","Data":"9bb51e6f0be213e7c49a19d3d8dbf471fa3de151bf6726e3c8d5e1c3b9d7bfab"} Oct 31 12:56:19.637701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:19.632486 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" event={"ID":"d2d18089-6846-4b29-bf2d-8b0d1e024b17","Type":"ContainerStarted","Data":"e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7"} Oct 31 12:56:19.637701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:19.632538 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" event={"ID":"d2d18089-6846-4b29-bf2d-8b0d1e024b17","Type":"ContainerStarted","Data":"06b96559c610dfc272b2cdef7c2785aeced60abc3b33a4c7ac816b5bb38a3f14"} Oct 31 12:56:19.662255 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:19.662196 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podStartSLOduration=1.662180835 podStartE2EDuration="1.662180835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:19.660994865 +0000 UTC m=+1473.838712851" watchObservedRunningTime="2026-10-31 12:56:19.662180835 +0000 UTC m=+1473.839898911" Oct 31 12:56:19.744660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:19.721789 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podStartSLOduration=2.72176767 podStartE2EDuration="2.72176767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:19.693501008 +0000 UTC m=+1473.871218994" watchObservedRunningTime="2026-10-31 12:56:19.72176767 +0000 UTC m=+1473.899485658" Oct 31 12:56:20.330656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:20.329018 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:20.334669 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:20.333778 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:20.334669 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:20.333868 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:20.418656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:20.417893 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:20.418656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:20.418044 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:20.418656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:20.418088 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:21.329076 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:21.329030 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:21.329608 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:21.329094 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:21.419658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:21.419050 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:21.419658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:21.419112 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:22.337607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.336787 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:22.337607 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.336846 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:22.418603 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.418565 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:22.418947 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.418903 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:22.662718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.658724 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 12:56:22.703824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.703774 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:22.784992 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.784938 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:22.839116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.839068 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:22.921832 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:22.921729 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:23.520320 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.499796 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:23.520866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.520439 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:23.520866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.520497 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:23.521203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.521095 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:23.521203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.521139 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:23.523803 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.523036 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 12:56:23.741307 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.693745 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:23.741307 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.694225 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:56:23.973779 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.946127 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:23.973779 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:23.947278 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:24.330712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.329815 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:24.330712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.329876 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:24.419660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.419542 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:24.419660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.419618 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:24.493657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.485389 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:24.513672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.500702 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:24.513672 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.502331 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:56:24.581388 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.566738 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:24.581388 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.567380 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:56:24.736866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:24.736812 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:25.083919 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.083232 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:56:25.108251 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.107734 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:25.169685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.166730 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:25.328336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.328301 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:25.361738 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.328604 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:25.370205 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.329046 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:25.374655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.369777 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:25.444658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.440595 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:25.448749 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.448704 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:25.500206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.500158 5074 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 31 12:56:25.502873 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.502832 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:25.740996 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.739716 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:25.776034 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.775989 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:56:25.861797 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:25.861752 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:26.144466 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.143770 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:26.225817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.187753 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:26.328654 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:56:26.317066 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:56:26.328654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.328042 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:26.328654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.328107 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:26.367752 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.366914 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:56:26.367957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.367934 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:26.385657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.382805 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 12:56:26.392809 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.392751 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 12:56:26.396749 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.396686 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:56:26.415775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.415745 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:56:26.443990 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.443939 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:26.444291 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.444246 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:26.462744 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.462700 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:56:26.463195 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:26.463176 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:56:27.328256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.328218 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 12:56:27.328781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.328275 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 12:56:27.415962 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.415922 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 12:56:27.416163 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.415992 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 12:56:27.480720 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.480682 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw"] Oct 31 12:56:27.491679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.491304 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.496645 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.496598 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:56:27.496924 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.496907 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:56:27.497199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.497183 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:56:27.497405 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.497391 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:56:27.507402 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.507369 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw"] Oct 31 12:56:27.530763 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.530735 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb"] Oct 31 12:56:27.551622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.551000 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.552483 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.552423 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbsb\" (UniqueName: \"kubernetes.io/projected/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-kube-api-access-dnbsb\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.552739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.552713 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.552814 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.552767 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.557710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.557688 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb"] Oct 31 12:56:27.653881 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.653786 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwkt\" (UniqueName: \"kubernetes.io/projected/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-kube-api-access-5bwkt\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.654173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.654142 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.654405 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.654375 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.654510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.654433 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.654510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.654469 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbsb\" (UniqueName: \"kubernetes.io/projected/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-kube-api-access-dnbsb\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.654671 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.654521 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.655482 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.655454 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.657840 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.657817 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.669866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.669835 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbsb\" (UniqueName: \"kubernetes.io/projected/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-kube-api-access-dnbsb\") pod \"maas-api-revoke-sub-1457cd2f-xg4hw\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.756608 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.756568 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.756944 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.756920 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwkt\" (UniqueName: \"kubernetes.io/projected/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-kube-api-access-5bwkt\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.757144 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.757127 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.757937 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.757904 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.760445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.760417 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.769839 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.769810 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwkt\" (UniqueName: \"kubernetes.io/projected/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-kube-api-access-5bwkt\") pod \"maas-api-revoke-sub-466b7fb5-p8wxb\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:27.830703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.830654 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:27.873675 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:27.873539 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:28.161990 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.161946 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw"] Oct 31 12:56:28.173112 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:28.173078 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612228c0_e5d3_4efc_ab43_f0b0b29bd5df.slice/crio-ac7b32a86383e3def1d0f010067f47049790a5398523931dde64c95c9ec78c31 WatchSource:0}: Error finding container ac7b32a86383e3def1d0f010067f47049790a5398523931dde64c95c9ec78c31: Status 404 returned error can't find the container with id ac7b32a86383e3def1d0f010067f47049790a5398523931dde64c95c9ec78c31 Oct 31 12:56:28.290450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.290399 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb"] Oct 31 12:56:28.432532 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.432423 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6"] Oct 31 12:56:28.441317 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.441290 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.445722 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.445598 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6"] Oct 31 12:56:28.477019 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.476980 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.477176 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.477056 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2lt\" (UniqueName: \"kubernetes.io/projected/e2e86576-8ab5-40db-9150-036471eb200b-kube-api-access-xg2lt\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.477176 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.477138 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.578442 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.578405 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2lt\" (UniqueName: \"kubernetes.io/projected/e2e86576-8ab5-40db-9150-036471eb200b-kube-api-access-xg2lt\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.578708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.578538 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.578849 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.578730 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.579576 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.579548 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.582894 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.582869 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.583684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.583610 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" event={"ID":"612228c0-e5d3-4efc-ab43-f0b0b29bd5df","Type":"ContainerStarted","Data":"ac7b32a86383e3def1d0f010067f47049790a5398523931dde64c95c9ec78c31"} Oct 31 12:56:28.586299 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.586276 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" event={"ID":"cac5cd21-89ad-4d47-ba77-898b4ee4ce59","Type":"ContainerStarted","Data":"1c515ad1999b27fbce275ecf6b49647f300b2937776631f663c5766299d3784d"} Oct 31 12:56:28.593153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.593126 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2lt\" (UniqueName: \"kubernetes.io/projected/e2e86576-8ab5-40db-9150-036471eb200b-kube-api-access-xg2lt\") pod \"maas-api-revoke-sub-3030e544-28qr6\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:28.671375 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.670323 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:28.671375 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.671309 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:28.672808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.672779 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 12:56:28.787501 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:28.787411 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:29.139592 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.139560 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6"] Oct 31 12:56:29.142228 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:29.142186 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e86576_8ab5_40db_9150_036471eb200b.slice/crio-58f0e15afec5f95d8a06c867918e88b2f0458164c558983730fc5b8d983407cf WatchSource:0}: Error finding container 58f0e15afec5f95d8a06c867918e88b2f0458164c558983730fc5b8d983407cf: Status 404 returned error can't find the container with id 58f0e15afec5f95d8a06c867918e88b2f0458164c558983730fc5b8d983407cf Oct 31 12:56:29.373178 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.372405 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:29.373178 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.372470 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:29.620798 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.618555 5074 generic.go:356] "Generic (PLEG): container finished" podID="612228c0-e5d3-4efc-ab43-f0b0b29bd5df" containerID="3e2aac30688b6b80a245e37224ab685cc548d0ca7a627241c3c50685f95dcaec" exitCode=0 Oct 31 12:56:29.620798 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.618670 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" event={"ID":"612228c0-e5d3-4efc-ab43-f0b0b29bd5df","Type":"ContainerDied","Data":"3e2aac30688b6b80a245e37224ab685cc548d0ca7a627241c3c50685f95dcaec"} Oct 31 12:56:29.631654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.621593 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" event={"ID":"e2e86576-8ab5-40db-9150-036471eb200b","Type":"ContainerStarted","Data":"58f0e15afec5f95d8a06c867918e88b2f0458164c558983730fc5b8d983407cf"} Oct 31 12:56:29.631654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.626395 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" event={"ID":"cac5cd21-89ad-4d47-ba77-898b4ee4ce59","Type":"ContainerStarted","Data":"beaf9531a4c2f116e0cd057c9ca4b879638534fe610f9115029b9e2a4b9f72c8"} Oct 31 12:56:29.665160 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.665064 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" podStartSLOduration=2.66505026 podStartE2EDuration="2.66505026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:29.660844432 +0000 UTC m=+1483.838562416" watchObservedRunningTime="2026-10-31 12:56:29.66505026 +0000 UTC m=+1483.842768273" Oct 31 12:56:29.795365 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.795327 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:29.797137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.797110 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:29.802902 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:29.802876 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 12:56:30.643245 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.643065 5074 generic.go:356] "Generic (PLEG): container finished" podID="cac5cd21-89ad-4d47-ba77-898b4ee4ce59" containerID="beaf9531a4c2f116e0cd057c9ca4b879638534fe610f9115029b9e2a4b9f72c8" exitCode=0 Oct 31 12:56:30.643245 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.643161 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" event={"ID":"cac5cd21-89ad-4d47-ba77-898b4ee4ce59","Type":"ContainerDied","Data":"beaf9531a4c2f116e0cd057c9ca4b879638534fe610f9115029b9e2a4b9f72c8"} Oct 31 12:56:30.645566 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.645533 5074 generic.go:356] "Generic (PLEG): container finished" podID="e2e86576-8ab5-40db-9150-036471eb200b" containerID="938b8adc1180de771f9cc5a578147905172e2ca935292d47784ab28488c09755" exitCode=0 Oct 31 12:56:30.646721 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.646616 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" event={"ID":"e2e86576-8ab5-40db-9150-036471eb200b","Type":"ContainerDied","Data":"938b8adc1180de771f9cc5a578147905172e2ca935292d47784ab28488c09755"} Oct 31 12:56:30.847872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.847833 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:30.969211 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.969176 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-kube-api-access-dnbsb\" (UniqueName: \"kubernetes.io/projected/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-kube-api-access-dnbsb\") pod \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " Oct 31 12:56:30.969389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.969267 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-xks-ca\") pod \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " Oct 31 12:56:30.969533 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.969507 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-openshift-ca\") pod \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\" (UID: \"612228c0-e5d3-4efc-ab43-f0b0b29bd5df\") " Oct 31 12:56:30.969935 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.969888 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-openshift-ca" pod "612228c0-e5d3-4efc-ab43-f0b0b29bd5df" (UID: "612228c0-e5d3-4efc-ab43-f0b0b29bd5df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:30.970561 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.970389 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:30.971552 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.971527 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-kube-api-access-dnbsb" pod "612228c0-e5d3-4efc-ab43-f0b0b29bd5df" (UID: "612228c0-e5d3-4efc-ab43-f0b0b29bd5df"). InnerVolumeSpecName "kube-api-access-dnbsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:30.971762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:30.971744 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-xks-ca" pod "612228c0-e5d3-4efc-ab43-f0b0b29bd5df" (UID: "612228c0-e5d3-4efc-ab43-f0b0b29bd5df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:31.072462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.072422 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnbsb\" (UniqueName: \"kubernetes.io/projected/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-kube-api-access-dnbsb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:31.072462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.072462 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612228c0-e5d3-4efc-ab43-f0b0b29bd5df-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:31.656554 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.656522 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" Oct 31 12:56:31.656554 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.656545 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw" event={"ID":"612228c0-e5d3-4efc-ab43-f0b0b29bd5df","Type":"ContainerDied","Data":"ac7b32a86383e3def1d0f010067f47049790a5398523931dde64c95c9ec78c31"} Oct 31 12:56:31.657146 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.656606 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7b32a86383e3def1d0f010067f47049790a5398523931dde64c95c9ec78c31" Oct 31 12:56:31.881035 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.881012 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:31.883547 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.883527 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:31.988611 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.988575 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-xks-ca\") pod \"e2e86576-8ab5-40db-9150-036471eb200b\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " Oct 31 12:56:31.988798 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.988689 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-kube-api-access-5bwkt\" (UniqueName: \"kubernetes.io/projected/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-kube-api-access-5bwkt\") pod \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " Oct 31 12:56:31.988798 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.988794 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e86576-8ab5-40db-9150-036471eb200b-kube-api-access-xg2lt\" (UniqueName: \"kubernetes.io/projected/e2e86576-8ab5-40db-9150-036471eb200b-kube-api-access-xg2lt\") pod \"e2e86576-8ab5-40db-9150-036471eb200b\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " Oct 31 12:56:31.989229 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.989201 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-openshift-ca\") pod \"e2e86576-8ab5-40db-9150-036471eb200b\" (UID: \"e2e86576-8ab5-40db-9150-036471eb200b\") " Oct 31 12:56:31.989619 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.989592 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-openshift-ca" pod "e2e86576-8ab5-40db-9150-036471eb200b" (UID: "e2e86576-8ab5-40db-9150-036471eb200b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:31.989861 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.989842 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-openshift-ca\") pod \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " Oct 31 12:56:31.990220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.990196 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-openshift-ca" pod "cac5cd21-89ad-4d47-ba77-898b4ee4ce59" (UID: "cac5cd21-89ad-4d47-ba77-898b4ee4ce59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:31.990326 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.990308 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-xks-ca\") pod \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\" (UID: \"cac5cd21-89ad-4d47-ba77-898b4ee4ce59\") " Oct 31 12:56:31.991158 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.991135 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:31.991254 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.991161 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:31.991322 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.991261 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-kube-api-access-5bwkt" pod "cac5cd21-89ad-4d47-ba77-898b4ee4ce59" (UID: "cac5cd21-89ad-4d47-ba77-898b4ee4ce59"). InnerVolumeSpecName "kube-api-access-5bwkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:31.991684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.991654 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-xks-ca" pod "e2e86576-8ab5-40db-9150-036471eb200b" (UID: "e2e86576-8ab5-40db-9150-036471eb200b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:31.992062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.992038 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e86576-8ab5-40db-9150-036471eb200b-kube-api-access-xg2lt" pod "e2e86576-8ab5-40db-9150-036471eb200b" (UID: "e2e86576-8ab5-40db-9150-036471eb200b"). InnerVolumeSpecName "kube-api-access-xg2lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:31.992555 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:31.992521 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-xks-ca" pod "cac5cd21-89ad-4d47-ba77-898b4ee4ce59" (UID: "cac5cd21-89ad-4d47-ba77-898b4ee4ce59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:32.093656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.092728 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:32.093656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.092768 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e86576-8ab5-40db-9150-036471eb200b-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:32.093656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.092782 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwkt\" (UniqueName: \"kubernetes.io/projected/cac5cd21-89ad-4d47-ba77-898b4ee4ce59-kube-api-access-5bwkt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:32.093656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.092797 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg2lt\" (UniqueName: \"kubernetes.io/projected/e2e86576-8ab5-40db-9150-036471eb200b-kube-api-access-xg2lt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:32.666887 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.666851 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" Oct 31 12:56:32.666887 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.666873 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6" event={"ID":"e2e86576-8ab5-40db-9150-036471eb200b","Type":"ContainerDied","Data":"58f0e15afec5f95d8a06c867918e88b2f0458164c558983730fc5b8d983407cf"} Oct 31 12:56:32.667396 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.666929 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f0e15afec5f95d8a06c867918e88b2f0458164c558983730fc5b8d983407cf" Oct 31 12:56:32.668466 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.668442 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" event={"ID":"cac5cd21-89ad-4d47-ba77-898b4ee4ce59","Type":"ContainerDied","Data":"1c515ad1999b27fbce275ecf6b49647f300b2937776631f663c5766299d3784d"} Oct 31 12:56:32.668466 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.668462 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb" Oct 31 12:56:32.668671 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:32.668481 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c515ad1999b27fbce275ecf6b49647f300b2937776631f663c5766299d3784d" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.786691 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7"] Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787547 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e2e86576-8ab5-40db-9150-036471eb200b" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787569 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e86576-8ab5-40db-9150-036471eb200b" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787590 5074 cpu_manager.go:414] "containerMap: removing container" podUID="612228c0-e5d3-4efc-ab43-f0b0b29bd5df" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787598 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612228c0-e5d3-4efc-ab43-f0b0b29bd5df" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787618 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cac5cd21-89ad-4d47-ba77-898b4ee4ce59" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787626 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac5cd21-89ad-4d47-ba77-898b4ee4ce59" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787862 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac5cd21-89ad-4d47-ba77-898b4ee4ce59" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787884 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="612228c0-e5d3-4efc-ab43-f0b0b29bd5df" containerName="revoke-keys" Oct 31 12:56:33.788708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.787893 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e86576-8ab5-40db-9150-036471eb200b" containerName="revoke-keys" Oct 31 12:56:33.793231 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.793205 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:33.804079 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.804054 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"distinct-w1-622e03-kserve-self-signed-certs\"" Oct 31 12:56:33.805122 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.805096 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7"] Oct 31 12:56:33.928353 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.928311 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-dshm\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:33.945458 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.945421 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-tmp-dir\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:33.946564 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.946535 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-home\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:33.971681 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.953728 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68176c7b-d2f0-425f-94e0-aadb0d173a37-tls-certs\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:33.978813 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.978767 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdfr5\" (UniqueName: \"kubernetes.io/projected/68176c7b-d2f0-425f-94e0-aadb0d173a37-kube-api-access-fdfr5\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:33.984815 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:33.984781 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-model-cache\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.086953 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-home\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.086999 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68176c7b-d2f0-425f-94e0-aadb0d173a37-tls-certs\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087270 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.087059 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfr5\" (UniqueName: \"kubernetes.io/projected/68176c7b-d2f0-425f-94e0-aadb0d173a37-kube-api-access-fdfr5\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087270 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.087093 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-model-cache\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087270 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.087158 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-dshm\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087270 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.087180 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-tmp-dir\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.087698 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-tmp-dir\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.087828 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.087764 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-home\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.088043 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.088022 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-model-cache\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.090427 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.090384 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-dshm\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.090953 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.090905 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68176c7b-d2f0-425f-94e0-aadb0d173a37-tls-certs\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.101199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.100902 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfr5\" (UniqueName: \"kubernetes.io/projected/68176c7b-d2f0-425f-94e0-aadb0d173a37-kube-api-access-fdfr5\") pod \"distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.110031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.110001 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:34.404581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.404547 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7"] Oct 31 12:56:34.405097 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:34.405067 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68176c7b_d2f0_425f_94e0_aadb0d173a37.slice/crio-3b2cbe8007147d461d111e816c8d68eee45f384ae5179c022ba3066f2b3f1c89 WatchSource:0}: Error finding container 3b2cbe8007147d461d111e816c8d68eee45f384ae5179c022ba3066f2b3f1c89: Status 404 returned error can't find the container with id 3b2cbe8007147d461d111e816c8d68eee45f384ae5179c022ba3066f2b3f1c89 Oct 31 12:56:34.478961 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.478938 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:34.492127 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.492089 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:34.706008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.705900 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" event={"ID":"68176c7b-d2f0-425f-94e0-aadb0d173a37","Type":"ContainerStarted","Data":"958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3"} Oct 31 12:56:34.706008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.705959 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" event={"ID":"68176c7b-d2f0-425f-94e0-aadb0d173a37","Type":"ContainerStarted","Data":"3b2cbe8007147d461d111e816c8d68eee45f384ae5179c022ba3066f2b3f1c89"} Oct 31 12:56:34.727726 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:34.727678 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podStartSLOduration=1.727664176 podStartE2EDuration="1.727664176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:34.726468856 +0000 UTC m=+1488.904186839" watchObservedRunningTime="2026-10-31 12:56:34.727664176 +0000 UTC m=+1488.905382159" Oct 31 12:56:35.110297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:35.110264 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:37.104103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.104064 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:56:37.717252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.717216 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72"] Oct 31 12:56:37.721600 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.721577 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.739689 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.739203 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:56:37.739689 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.739335 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:56:37.739689 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.739423 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:56:37.742789 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.740622 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:56:37.752651 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.748876 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72"] Oct 31 12:56:37.863653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.860397 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.863653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.860533 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66btq\" (UniqueName: \"kubernetes.io/projected/6ec324f3-c445-4548-a102-8ea170d3d269-kube-api-access-66btq\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.863653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.860690 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.961983 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.961936 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66btq\" (UniqueName: \"kubernetes.io/projected/6ec324f3-c445-4548-a102-8ea170d3d269-kube-api-access-66btq\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.962400 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.962376 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.962622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.962602 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.964112 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.963901 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:37.993744 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:37.991166 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:38.017655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.009705 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66btq\" (UniqueName: \"kubernetes.io/projected/6ec324f3-c445-4548-a102-8ea170d3d269-kube-api-access-66btq\") pod \"maas-api-revoke-sub-2f2db079-msh72\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:38.075175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.046759 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7"] Oct 31 12:56:38.075175 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.062830 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:38.094903 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.080957 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7"] Oct 31 12:56:38.094903 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.081112 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.237724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.215141 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.237724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.215195 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94gfr\" (UniqueName: \"kubernetes.io/projected/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-kube-api-access-94gfr\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.237724 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.215243 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.338677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.329395 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.338677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.329445 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94gfr\" (UniqueName: \"kubernetes.io/projected/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-kube-api-access-94gfr\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.338677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.329492 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.340262 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.339543 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.389837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.389764 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.457081 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.438455 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94gfr\" (UniqueName: \"kubernetes.io/projected/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-kube-api-access-94gfr\") pod \"maas-api-revoke-sub-c8df54bb-9txv7\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:38.713705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:38.710048 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:39.225558 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.225510 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.488750 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.488651 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 12:56:39.518777 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.518730 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.684658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.672154 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.874866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.874755 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.928662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.915797 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:39.943666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.928890 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 12:56:39.943666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:39.929299 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 12:56:40.096471 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.091887 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.210759 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.191819 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72"] Oct 31 12:56:40.230664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.228759 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.230664 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.229571 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:40.413712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.412788 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.498024 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.482751 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.550655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.549746 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.555795 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.555753 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.556441 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.556417 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:40.627104 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.627060 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.657710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.648903 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:56:40.909910 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.909860 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:40.914657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:40.910806 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:56:41.031668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.029765 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7"] Oct 31 12:56:41.031668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.029932 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 12:56:41.045737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.032155 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.045737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.033827 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:56:41.045737 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.033997 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:56:41.519682 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.506529 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.519682 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.509115 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:56:41.519682 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.509767 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.621684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.598608 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 12:56:41.621684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.600044 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:56:41.727195 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.726731 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:41.727195 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.726817 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.796795 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.770017 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.796795 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.772415 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.796795 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.783420 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 12:56:41.845589 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.845536 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:41.918906 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.918859 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 31 12:56:41.943569 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.801248 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:56:41.996894 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:41.996859 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:42.004781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.004736 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.031008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.030956 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.038708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.034868 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.361005 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.352021 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.507712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.506735 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:42.507712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.506805 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.507712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.506890 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.511120 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.511078 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.644683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.585833 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" event={"ID":"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e","Type":"ContainerStarted","Data":"ff075eb0eda573225769564bc0e00b97b0dbedeba42b388660597ed6354175b1"} Oct 31 12:56:42.644683 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.614971 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.668100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.668057 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" event={"ID":"6ec324f3-c445-4548-a102-8ea170d3d269","Type":"ContainerStarted","Data":"d9119294778bd3cddc3ebf9002a2a9134c2f36f32ab9b2259cd3062d239a91df"} Oct 31 12:56:42.755702 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.670456 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.837870 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.786201 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.847739 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.786290 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.852657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.794441 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:42.862760 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:42.794502 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.052758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.052670 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:56:43.053020 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.052987 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.172460 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.172420 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 31 12:56:43.226945 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.172710 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.226945 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.172759 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.475776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.395069 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 12:56:43.475776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.395735 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:56:43.508420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.507840 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.521325 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.521025 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:43.557871 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.557775 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 12:56:43.765659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.756247 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 12:56:43.862762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.839143 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 12:56:43.862762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.854802 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 12:56:43.907757 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:43.904430 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:44.135000 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.134969 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:56:44.135221 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.135207 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:44.135380 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.135365 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:56:44.159667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.158957 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" event={"ID":"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e","Type":"ContainerStarted","Data":"a0356c07f028bdf0879b7b40afe84a169f71ece5353df0f090bd7373a9e57aef"} Oct 31 12:56:44.232698 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.230408 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:44.372971 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.372882 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 12:56:44.391456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.391431 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 12:56:44.527289 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:56:44.520494 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bcf95d_8be5_4343_a9c2_3f2ab5ec519e.slice/crio-conmon-a0356c07f028bdf0879b7b40afe84a169f71ece5353df0f090bd7373a9e57aef.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec324f3_c445_4548_a102_8ea170d3d269.slice/crio-c9c75a050e6815f30a87815c295560ef8763fd519f0a41401bf55f6cb1eaee03.scope\": RecentStats: unable to find data in memory cache]" Oct 31 12:56:44.542436 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.542402 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" event={"ID":"6ec324f3-c445-4548-a102-8ea170d3d269","Type":"ContainerStarted","Data":"c9c75a050e6815f30a87815c295560ef8763fd519f0a41401bf55f6cb1eaee03"} Oct 31 12:56:44.549611 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.549574 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 12:56:44.559670 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.559143 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" podStartSLOduration=7.559123978 podStartE2EDuration="7.559123978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:44.386132605 +0000 UTC m=+1498.563850617" watchObservedRunningTime="2026-10-31 12:56:44.559123978 +0000 UTC m=+1498.736841966" Oct 31 12:56:44.674707 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.674564 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" podStartSLOduration=7.674545075 podStartE2EDuration="7.674545075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:44.621151329 +0000 UTC m=+1498.798869318" watchObservedRunningTime="2026-10-31 12:56:44.674545075 +0000 UTC m=+1498.852263060" Oct 31 12:56:44.760656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.757312 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:44.828704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.827167 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:56:44.853045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.853016 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:44.855036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.855015 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 12:56:44.881524 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.881484 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:44.955748 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.955667 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:44.989468 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:44.985363 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:56:45.061957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:45.060479 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:56:45.134811 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:45.134732 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:45.550550 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:45.550512 5074 generic.go:356] "Generic (PLEG): container finished" podID="6ec324f3-c445-4548-a102-8ea170d3d269" containerID="c9c75a050e6815f30a87815c295560ef8763fd519f0a41401bf55f6cb1eaee03" exitCode=0 Oct 31 12:56:45.550885 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:45.550562 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" event={"ID":"6ec324f3-c445-4548-a102-8ea170d3d269","Type":"ContainerDied","Data":"c9c75a050e6815f30a87815c295560ef8763fd519f0a41401bf55f6cb1eaee03"} Oct 31 12:56:45.552421 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:45.552385 5074 generic.go:356] "Generic (PLEG): container finished" podID="a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" containerID="a0356c07f028bdf0879b7b40afe84a169f71ece5353df0f090bd7373a9e57aef" exitCode=0 Oct 31 12:56:45.552551 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:45.552438 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" event={"ID":"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e","Type":"ContainerDied","Data":"a0356c07f028bdf0879b7b40afe84a169f71ece5353df0f090bd7373a9e57aef"} Oct 31 12:56:46.141911 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.141880 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:56:46.152170 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.152147 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 12:56:46.798153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.798119 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:46.801313 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.801290 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:46.964464 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.964385 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-kube-api-access-94gfr\" (UniqueName: \"kubernetes.io/projected/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-kube-api-access-94gfr\") pod \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " Oct 31 12:56:46.964464 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.964447 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-xks-ca\") pod \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " Oct 31 12:56:46.964762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.964567 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-xks-ca\") pod \"6ec324f3-c445-4548-a102-8ea170d3d269\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " Oct 31 12:56:46.964762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.964599 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-openshift-ca\") pod \"6ec324f3-c445-4548-a102-8ea170d3d269\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " Oct 31 12:56:46.964762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.964657 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec324f3-c445-4548-a102-8ea170d3d269-kube-api-access-66btq\" (UniqueName: \"kubernetes.io/projected/6ec324f3-c445-4548-a102-8ea170d3d269-kube-api-access-66btq\") pod \"6ec324f3-c445-4548-a102-8ea170d3d269\" (UID: \"6ec324f3-c445-4548-a102-8ea170d3d269\") " Oct 31 12:56:46.964762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.964730 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-openshift-ca\") pod \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\" (UID: \"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e\") " Oct 31 12:56:46.965010 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.964980 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-openshift-ca" pod "6ec324f3-c445-4548-a102-8ea170d3d269" (UID: "6ec324f3-c445-4548-a102-8ea170d3d269"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:46.965132 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.965102 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-openshift-ca" pod "a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" (UID: "a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:46.965256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.965223 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:46.965256 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.965243 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:46.966888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.966866 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-xks-ca" pod "6ec324f3-c445-4548-a102-8ea170d3d269" (UID: "6ec324f3-c445-4548-a102-8ea170d3d269"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:46.966888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.966875 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-xks-ca" pod "a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" (UID: "a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:46.967149 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.967132 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec324f3-c445-4548-a102-8ea170d3d269-kube-api-access-66btq" pod "6ec324f3-c445-4548-a102-8ea170d3d269" (UID: "6ec324f3-c445-4548-a102-8ea170d3d269"). InnerVolumeSpecName "kube-api-access-66btq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:46.967198 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:46.967183 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-kube-api-access-94gfr" pod "a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" (UID: "a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e"). InnerVolumeSpecName "kube-api-access-94gfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:47.066462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.066419 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec324f3-c445-4548-a102-8ea170d3d269-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:47.066462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.066462 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66btq\" (UniqueName: \"kubernetes.io/projected/6ec324f3-c445-4548-a102-8ea170d3d269-kube-api-access-66btq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:47.066710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.066479 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94gfr\" (UniqueName: \"kubernetes.io/projected/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-kube-api-access-94gfr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:47.066710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.066492 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:47.582009 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.581957 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" Oct 31 12:56:47.582009 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.581983 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7" event={"ID":"a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e","Type":"ContainerDied","Data":"ff075eb0eda573225769564bc0e00b97b0dbedeba42b388660597ed6354175b1"} Oct 31 12:56:47.582563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.582044 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff075eb0eda573225769564bc0e00b97b0dbedeba42b388660597ed6354175b1" Oct 31 12:56:47.584028 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.584001 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" event={"ID":"6ec324f3-c445-4548-a102-8ea170d3d269","Type":"ContainerDied","Data":"d9119294778bd3cddc3ebf9002a2a9134c2f36f32ab9b2259cd3062d239a91df"} Oct 31 12:56:47.584169 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.584050 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9119294778bd3cddc3ebf9002a2a9134c2f36f32ab9b2259cd3062d239a91df" Oct 31 12:56:47.584169 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:47.584054 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72" Oct 31 12:56:51.914818 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:51.913703 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 12:56:52.107904 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:52.107831 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.122605 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:52.122552 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.163330 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:52.162745 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.420667 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:52.410742 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.515659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:52.514756 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.515659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:52.515282 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:52.887656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:52.886720 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 12:56:53.144177 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.143736 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 12:56:53.165492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.164799 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:53.172728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.171001 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:56:53.172728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.171069 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:53.522787 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.522743 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:56:53.523842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.523813 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:53.560921 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.560889 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:56:53.837837 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.837748 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:56:53.920745 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:53.920713 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:56:55.251872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.251837 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7"] Oct 31 12:56:55.252395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.252381 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6ec324f3-c445-4548-a102-8ea170d3d269" containerName="revoke-keys" Oct 31 12:56:55.252439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.252396 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec324f3-c445-4548-a102-8ea170d3d269" containerName="revoke-keys" Oct 31 12:56:55.252439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.252407 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" containerName="revoke-keys" Oct 31 12:56:55.252439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.252412 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" containerName="revoke-keys" Oct 31 12:56:55.252530 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.252518 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" containerName="revoke-keys" Oct 31 12:56:55.252562 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.252531 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec324f3-c445-4548-a102-8ea170d3d269" containerName="revoke-keys" Oct 31 12:56:55.257728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.257708 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.261762 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.261741 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:56:55.262499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.262481 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:56:55.262707 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.262517 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:56:55.262778 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.262516 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:56:55.277511 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.269072 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7"] Oct 31 12:56:55.315450 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.315411 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqql\" (UniqueName: \"kubernetes.io/projected/51f334b4-7f23-4f02-99e6-677095e15c62-kube-api-access-chqql\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.315613 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.315564 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.315725 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.315689 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.416869 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.416830 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqql\" (UniqueName: \"kubernetes.io/projected/51f334b4-7f23-4f02-99e6-677095e15c62-kube-api-access-chqql\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.417061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.416917 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.417061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.416980 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.417669 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.417606 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.419570 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.419536 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.429318 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.429292 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqql\" (UniqueName: \"kubernetes.io/projected/51f334b4-7f23-4f02-99e6-677095e15c62-kube-api-access-chqql\") pod \"maas-api-revoke-sub-7a2af416-z77v7\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.579157 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.579075 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:55.866144 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:55.866111 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7"] Oct 31 12:56:55.866616 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:56:55.866580 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f334b4_7f23_4f02_99e6_677095e15c62.slice/crio-ffe15cc42015bb065b63026e2278de0f09c4a9557b060d68f4eb1b1d86ba67b3 WatchSource:0}: Error finding container ffe15cc42015bb065b63026e2278de0f09c4a9557b060d68f4eb1b1d86ba67b3: Status 404 returned error can't find the container with id ffe15cc42015bb065b63026e2278de0f09c4a9557b060d68f4eb1b1d86ba67b3 Oct 31 12:56:55.992609 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:56:55.992565 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:56:56.779677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:56.779612 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" event={"ID":"51f334b4-7f23-4f02-99e6-677095e15c62","Type":"ContainerStarted","Data":"33dde7601876dc04f5c47129a89a8ae38e7523b4594592027d31cbdabb43318e"} Oct 31 12:56:56.780179 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:56.779696 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" event={"ID":"51f334b4-7f23-4f02-99e6-677095e15c62","Type":"ContainerStarted","Data":"ffe15cc42015bb065b63026e2278de0f09c4a9557b060d68f4eb1b1d86ba67b3"} Oct 31 12:56:56.797878 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:56.797827 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" podStartSLOduration=1.7978135910000002 podStartE2EDuration="1.797813591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:56.797605676 +0000 UTC m=+1510.975323681" watchObservedRunningTime="2026-10-31 12:56:56.797813591 +0000 UTC m=+1510.975531574" Oct 31 12:56:57.790735 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:57.790679 5074 generic.go:356] "Generic (PLEG): container finished" podID="51f334b4-7f23-4f02-99e6-677095e15c62" containerID="33dde7601876dc04f5c47129a89a8ae38e7523b4594592027d31cbdabb43318e" exitCode=0 Oct 31 12:56:57.791158 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:57.790789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" event={"ID":"51f334b4-7f23-4f02-99e6-677095e15c62","Type":"ContainerDied","Data":"33dde7601876dc04f5c47129a89a8ae38e7523b4594592027d31cbdabb43318e"} Oct 31 12:56:58.963676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:58.963626 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:59.061934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.061852 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-xks-ca\") pod \"51f334b4-7f23-4f02-99e6-677095e15c62\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " Oct 31 12:56:59.062107 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.061943 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f334b4-7f23-4f02-99e6-677095e15c62-kube-api-access-chqql\" (UniqueName: \"kubernetes.io/projected/51f334b4-7f23-4f02-99e6-677095e15c62-kube-api-access-chqql\") pod \"51f334b4-7f23-4f02-99e6-677095e15c62\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " Oct 31 12:56:59.062107 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.062022 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-openshift-ca\") pod \"51f334b4-7f23-4f02-99e6-677095e15c62\" (UID: \"51f334b4-7f23-4f02-99e6-677095e15c62\") " Oct 31 12:56:59.062385 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.062352 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-openshift-ca" pod "51f334b4-7f23-4f02-99e6-677095e15c62" (UID: "51f334b4-7f23-4f02-99e6-677095e15c62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:56:59.062707 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.062686 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:59.064226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.064195 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f334b4-7f23-4f02-99e6-677095e15c62-kube-api-access-chqql" pod "51f334b4-7f23-4f02-99e6-677095e15c62" (UID: "51f334b4-7f23-4f02-99e6-677095e15c62"). InnerVolumeSpecName "kube-api-access-chqql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:56:59.064419 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.064398 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-xks-ca" pod "51f334b4-7f23-4f02-99e6-677095e15c62" (UID: "51f334b4-7f23-4f02-99e6-677095e15c62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:56:59.164305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.164271 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f334b4-7f23-4f02-99e6-677095e15c62-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:59.164305 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.164303 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chqql\" (UniqueName: \"kubernetes.io/projected/51f334b4-7f23-4f02-99e6-677095e15c62-kube-api-access-chqql\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:56:59.809348 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.809316 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" Oct 31 12:56:59.809348 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.809334 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7" event={"ID":"51f334b4-7f23-4f02-99e6-677095e15c62","Type":"ContainerDied","Data":"ffe15cc42015bb065b63026e2278de0f09c4a9557b060d68f4eb1b1d86ba67b3"} Oct 31 12:56:59.809599 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.809434 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe15cc42015bb065b63026e2278de0f09c4a9557b060d68f4eb1b1d86ba67b3" Oct 31 12:56:59.890701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.890665 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn"] Oct 31 12:56:59.891756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.891711 5074 cpu_manager.go:414] "containerMap: removing container" podUID="51f334b4-7f23-4f02-99e6-677095e15c62" containerName="revoke-keys" Oct 31 12:56:59.891756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.891743 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f334b4-7f23-4f02-99e6-677095e15c62" containerName="revoke-keys" Oct 31 12:56:59.891981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.891965 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f334b4-7f23-4f02-99e6-677095e15c62" containerName="revoke-keys" Oct 31 12:56:59.905585 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.905556 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.911310 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.911283 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn"] Oct 31 12:56:59.974115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974076 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfrg\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-kube-api-access-gdfrg\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974124 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974225 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974261 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974318 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974342 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974590 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974367 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974617 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:56:59.974981 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:56:59.974716 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076266 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076179 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfrg\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-kube-api-access-gdfrg\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076266 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076238 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076288 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076309 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076339 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076363 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076398 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076603 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076663 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076722 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076756 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.076977 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.076908 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.077085 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.077064 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.077391 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.077365 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.079143 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.079115 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.079424 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.079406 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.091364 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.091340 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfrg\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-kube-api-access-gdfrg\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.091648 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.091619 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.225759 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.225727 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:00.376453 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.372272 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:57:00.528952 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.528921 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn"] Oct 31 12:57:00.529229 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:57:00.529204 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38504a5_0bbf_485f_9a06_5e9964ff7b88.slice/crio-3c6a7db5f5753c589082777aa752ef01cde341fb098c0e873edd21679f0a0756 WatchSource:0}: Error finding container 3c6a7db5f5753c589082777aa752ef01cde341fb098c0e873edd21679f0a0756: Status 404 returned error can't find the container with id 3c6a7db5f5753c589082777aa752ef01cde341fb098c0e873edd21679f0a0756 Oct 31 12:57:00.532061 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.532027 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:00.532137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.532100 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:00.532137 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.532131 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:00.821998 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.821949 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" event={"ID":"e38504a5-0bbf-485f-9a06-5e9964ff7b88","Type":"ContainerStarted","Data":"ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae"} Oct 31 12:57:00.822153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.822026 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" event={"ID":"e38504a5-0bbf-485f-9a06-5e9964ff7b88","Type":"ContainerStarted","Data":"3c6a7db5f5753c589082777aa752ef01cde341fb098c0e873edd21679f0a0756"} Oct 31 12:57:00.848973 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:00.848916 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podStartSLOduration=1.848902846 podStartE2EDuration="1.848902846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:00.848454749 +0000 UTC m=+1515.026172736" watchObservedRunningTime="2026-10-31 12:57:00.848902846 +0000 UTC m=+1515.026620830" Oct 31 12:57:01.233395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:01.227664 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:01.233395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:01.228499 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:01.233395 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:01.228540 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:02.227131 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:02.227095 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:02.227925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:02.227474 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:02.512082 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:02.511979 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 12:57:02.733813 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:02.733750 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:03.229147 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:03.229091 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:03.229357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:03.229319 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:04.241846 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:04.241810 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:04.242453 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:04.242418 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:04.571715 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:04.571413 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:04.864308 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:04.864205 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:04.864589 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:04.864549 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:05.235625 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.235373 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:05.235625 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.235437 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:05.453669 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.450796 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:05.549735 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.549651 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c"] Oct 31 12:57:05.559878 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.559847 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c"] Oct 31 12:57:05.561783 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.560900 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.583017 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.582692 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsjt\" (UniqueName: \"kubernetes.io/projected/474314d9-4d48-49b2-96fc-7cf8e1765a47-kube-api-access-8dsjt\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.583017 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.582758 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.583017 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.582785 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.583743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.583495 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:57:05.583743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.583584 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:57:05.583743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.583655 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:57:05.583743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.583712 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:57:05.695659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.694284 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsjt\" (UniqueName: \"kubernetes.io/projected/474314d9-4d48-49b2-96fc-7cf8e1765a47-kube-api-access-8dsjt\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.695659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.694350 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.695659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.694376 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.704190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.704161 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.758765 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.758734 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsjt\" (UniqueName: \"kubernetes.io/projected/474314d9-4d48-49b2-96fc-7cf8e1765a47-kube-api-access-8dsjt\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.760431 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.760407 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af331fab-59x6c\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:05.782753 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:05.782725 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:06.341656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.313054 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:06.341656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.313115 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:06.430943 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.430852 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c"] Oct 31 12:57:06.565610 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.565559 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb"] Oct 31 12:57:06.581043 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:57:06.580994 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474314d9_4d48_49b2_96fc_7cf8e1765a47.slice/crio-5e61fb998b084ebc1a331d87cecfceedd77e07e77f672843ed3e0acc058dbff2 WatchSource:0}: Error finding container 5e61fb998b084ebc1a331d87cecfceedd77e07e77f672843ed3e0acc058dbff2: Status 404 returned error can't find the container with id 5e61fb998b084ebc1a331d87cecfceedd77e07e77f672843ed3e0acc058dbff2 Oct 31 12:57:06.626746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.623812 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb"] Oct 31 12:57:06.626746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.623934 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.630752 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.630717 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-home\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.631937 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.631895 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-dshm\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.632142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.632110 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4qf\" (UniqueName: \"kubernetes.io/projected/4c4dc8be-7585-47cc-a538-cd3d332eee43-kube-api-access-2h4qf\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.632329 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.632310 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4dc8be-7585-47cc-a538-cd3d332eee43-tls-certs\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.640172 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.635526 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-622e03\"/\"distinct-2-w1-622e03-kserve-self-signed-certs\"" Oct 31 12:57:06.640726 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.640702 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-model-cache\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.640936 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.640917 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-tmp-dir\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.766042 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.765848 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-tmp-dir\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.766433 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.766409 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-home\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.766594 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.766578 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-dshm\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.766731 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.766715 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4qf\" (UniqueName: \"kubernetes.io/projected/4c4dc8be-7585-47cc-a538-cd3d332eee43-kube-api-access-2h4qf\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.766877 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.766863 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4dc8be-7585-47cc-a538-cd3d332eee43-tls-certs\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.767012 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.766996 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-model-cache\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.767838 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.767815 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-model-cache\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.768258 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.768233 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-tmp-dir\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.768653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.768612 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-home\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.934230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.934200 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-dshm\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.934998 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.934979 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4dc8be-7585-47cc-a538-cd3d332eee43-tls-certs\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:06.936519 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:06.936489 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4qf\" (UniqueName: \"kubernetes.io/projected/4c4dc8be-7585-47cc-a538-cd3d332eee43-kube-api-access-2h4qf\") pod \"distinct-2-w1-622e03-kserve-c69d8bf5-skmxb\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:07.015558 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:07.015517 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" event={"ID":"474314d9-4d48-49b2-96fc-7cf8e1765a47","Type":"ContainerStarted","Data":"5e61fb998b084ebc1a331d87cecfceedd77e07e77f672843ed3e0acc058dbff2"} Oct 31 12:57:07.097655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:07.093319 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:07.227263 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:07.226897 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:07.227263 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:07.226946 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:07.434154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:07.434105 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb"] Oct 31 12:57:08.030594 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.030554 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" event={"ID":"4c4dc8be-7585-47cc-a538-cd3d332eee43","Type":"ContainerStarted","Data":"75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194"} Oct 31 12:57:08.031088 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.030608 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" event={"ID":"4c4dc8be-7585-47cc-a538-cd3d332eee43","Type":"ContainerStarted","Data":"c2b9da674d4b44a98c9c1c68f0218b1fa751f22b5de5c7f125f4f4e521c5a06c"} Oct 31 12:57:08.032292 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.032264 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" event={"ID":"474314d9-4d48-49b2-96fc-7cf8e1765a47","Type":"ContainerStarted","Data":"5163a82f95fd5c437f95d8d6ffc2821ace660d60cdb1a4f390705cecf5236f8a"} Oct 31 12:57:08.053092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.053048 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podStartSLOduration=2.053035989 podStartE2EDuration="2.053035989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:08.050447337 +0000 UTC m=+1522.228165323" watchObservedRunningTime="2026-10-31 12:57:08.053035989 +0000 UTC m=+1522.230753998" Oct 31 12:57:08.094910 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.094828 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:08.107676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.103304 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" podStartSLOduration=3.103283129 podStartE2EDuration="3.103283129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:08.067790804 +0000 UTC m=+1522.245508788" watchObservedRunningTime="2026-10-31 12:57:08.103283129 +0000 UTC m=+1522.281001115" Oct 31 12:57:08.226444 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.226402 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:08.226622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:08.226503 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:09.042453 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:09.042415 5074 generic.go:356] "Generic (PLEG): container finished" podID="474314d9-4d48-49b2-96fc-7cf8e1765a47" containerID="5163a82f95fd5c437f95d8d6ffc2821ace660d60cdb1a4f390705cecf5236f8a" exitCode=0 Oct 31 12:57:09.042453 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:09.042449 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" event={"ID":"474314d9-4d48-49b2-96fc-7cf8e1765a47","Type":"ContainerDied","Data":"5163a82f95fd5c437f95d8d6ffc2821ace660d60cdb1a4f390705cecf5236f8a"} Oct 31 12:57:09.226136 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:09.226100 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 12:57:09.226302 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:09.226167 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 12:57:10.254614 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.254583 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:10.359989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.359900 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-openshift-ca\") pod \"474314d9-4d48-49b2-96fc-7cf8e1765a47\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " Oct 31 12:57:10.359989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.359975 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-xks-ca\") pod \"474314d9-4d48-49b2-96fc-7cf8e1765a47\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " Oct 31 12:57:10.360199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.360044 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474314d9-4d48-49b2-96fc-7cf8e1765a47-kube-api-access-8dsjt\" (UniqueName: \"kubernetes.io/projected/474314d9-4d48-49b2-96fc-7cf8e1765a47-kube-api-access-8dsjt\") pod \"474314d9-4d48-49b2-96fc-7cf8e1765a47\" (UID: \"474314d9-4d48-49b2-96fc-7cf8e1765a47\") " Oct 31 12:57:10.360332 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.360298 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-openshift-ca" pod "474314d9-4d48-49b2-96fc-7cf8e1765a47" (UID: "474314d9-4d48-49b2-96fc-7cf8e1765a47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:57:10.360674 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.360651 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:10.362316 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.362292 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-xks-ca" pod "474314d9-4d48-49b2-96fc-7cf8e1765a47" (UID: "474314d9-4d48-49b2-96fc-7cf8e1765a47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:57:10.362406 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.362355 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474314d9-4d48-49b2-96fc-7cf8e1765a47-kube-api-access-8dsjt" pod "474314d9-4d48-49b2-96fc-7cf8e1765a47" (UID: "474314d9-4d48-49b2-96fc-7cf8e1765a47"). InnerVolumeSpecName "kube-api-access-8dsjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:57:10.461892 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.461852 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474314d9-4d48-49b2-96fc-7cf8e1765a47-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:10.461892 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:10.461880 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsjt\" (UniqueName: \"kubernetes.io/projected/474314d9-4d48-49b2-96fc-7cf8e1765a47-kube-api-access-8dsjt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:11.059830 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.059793 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" Oct 31 12:57:11.059830 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.059803 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c" event={"ID":"474314d9-4d48-49b2-96fc-7cf8e1765a47","Type":"ContainerDied","Data":"5e61fb998b084ebc1a331d87cecfceedd77e07e77f672843ed3e0acc058dbff2"} Oct 31 12:57:11.060062 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.059870 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e61fb998b084ebc1a331d87cecfceedd77e07e77f672843ed3e0acc058dbff2" Oct 31 12:57:11.281401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.280790 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:11.281401 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.280853 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:11.593416 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.593380 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:11.593822 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.593800 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:11.594757 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:11.594700 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 12:57:15.903173 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.903139 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl"] Oct 31 12:57:15.904008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.903989 5074 cpu_manager.go:414] "containerMap: removing container" podUID="474314d9-4d48-49b2-96fc-7cf8e1765a47" containerName="revoke-keys" Oct 31 12:57:15.904069 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.904010 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474314d9-4d48-49b2-96fc-7cf8e1765a47" containerName="revoke-keys" Oct 31 12:57:15.904210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.904198 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="474314d9-4d48-49b2-96fc-7cf8e1765a47" containerName="revoke-keys" Oct 31 12:57:15.909579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.909531 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:15.914810 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.914782 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-702419-dockercfg-695tv\"" Oct 31 12:57:15.914932 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.914828 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-702419\"" Oct 31 12:57:15.917370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.917343 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl"] Oct 31 12:57:15.943426 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.943397 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5"] Oct 31 12:57:15.947428 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.947405 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:15.948047 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.948013 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc6f7\" (UniqueName: \"kubernetes.io/projected/53634ad4-669e-4e07-96be-a14aa9c23543-kube-api-access-gc6f7\") pod \"payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:15.948152 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.948084 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53634ad4-669e-4e07-96be-a14aa9c23543-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:15.952106 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:15.952083 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5"] Oct 31 12:57:16.060669 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.049608 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc6f7\" (UniqueName: \"kubernetes.io/projected/53634ad4-669e-4e07-96be-a14aa9c23543-kube-api-access-gc6f7\") pod \"payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:16.098843 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.098802 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53634ad4-669e-4e07-96be-a14aa9c23543-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:16.100511 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.100329 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53634ad4-669e-4e07-96be-a14aa9c23543-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:16.124696 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.117181 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc6f7\" (UniqueName: \"kubernetes.io/projected/53634ad4-669e-4e07-96be-a14aa9c23543-kube-api-access-gc6f7\") pod \"payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:16.205543 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.205450 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx99v\" (UniqueName: \"kubernetes.io/projected/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-kube-api-access-gx99v\") pod \"payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:16.205735 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.205620 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:16.228651 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.228604 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:16.316416 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.306957 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx99v\" (UniqueName: \"kubernetes.io/projected/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-kube-api-access-gx99v\") pod \"payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:16.316416 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.307083 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:16.316416 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.311751 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:16.323679 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.321079 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx99v\" (UniqueName: \"kubernetes.io/projected/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-kube-api-access-gx99v\") pod \"payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:16.517024 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:57:16.516985 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53634ad4_669e_4e07_96be_a14aa9c23543.slice/crio-192c30da59227b546b31db0d2fb4faaee7ff22b372f00cf56c2bf63e95678d88 WatchSource:0}: Error finding container 192c30da59227b546b31db0d2fb4faaee7ff22b372f00cf56c2bf63e95678d88: Status 404 returned error can't find the container with id 192c30da59227b546b31db0d2fb4faaee7ff22b372f00cf56c2bf63e95678d88 Oct 31 12:57:16.517292 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.517268 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl"] Oct 31 12:57:16.571897 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.566431 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:16.858570 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:57:16.858529 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79246ba3_75f6_4afa_9ac0_9448e5bdf6f2.slice/crio-72fae9edd7ab9d65eb9ed7e1736c80c9b99d92a33b7e03ec452d0c13bd23c343 WatchSource:0}: Error finding container 72fae9edd7ab9d65eb9ed7e1736c80c9b99d92a33b7e03ec452d0c13bd23c343: Status 404 returned error can't find the container with id 72fae9edd7ab9d65eb9ed7e1736c80c9b99d92a33b7e03ec452d0c13bd23c343 Oct 31 12:57:16.858749 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:16.858683 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5"] Oct 31 12:57:17.094657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.093950 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:17.122399 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.122315 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:17.173508 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.172187 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" event={"ID":"53634ad4-669e-4e07-96be-a14aa9c23543","Type":"ContainerStarted","Data":"ae00cad027a5cfee827e3c46f87de1df545251acc4dc684c817d2a3c4253e42e"} Oct 31 12:57:17.173508 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.172238 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" event={"ID":"53634ad4-669e-4e07-96be-a14aa9c23543","Type":"ContainerStarted","Data":"192c30da59227b546b31db0d2fb4faaee7ff22b372f00cf56c2bf63e95678d88"} Oct 31 12:57:17.173508 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.172516 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:17.192896 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.191173 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" event={"ID":"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2","Type":"ContainerStarted","Data":"41139cf5a3be546c36e7d01332212bc3d16526a454a11bebfbd8da74394cc02c"} Oct 31 12:57:17.192896 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.191335 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:17.192896 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.191367 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" event={"ID":"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2","Type":"ContainerStarted","Data":"72fae9edd7ab9d65eb9ed7e1736c80c9b99d92a33b7e03ec452d0c13bd23c343"} Oct 31 12:57:17.201860 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.198991 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" podStartSLOduration=2.198972896 podStartE2EDuration="2.198972896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:17.191021286 +0000 UTC m=+1531.368739275" watchObservedRunningTime="2026-10-31 12:57:17.198972896 +0000 UTC m=+1531.376690881" Oct 31 12:57:17.205779 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.205754 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:17.217074 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:17.217028 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" podStartSLOduration=2.217016444 podStartE2EDuration="2.217016444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:17.215091755 +0000 UTC m=+1531.392809747" watchObservedRunningTime="2026-10-31 12:57:17.217016444 +0000 UTC m=+1531.394734427" Oct 31 12:57:19.384586 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.384546 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq"] Oct 31 12:57:19.389788 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.389758 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.402510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.402487 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-622e03-openshift-default-dockercfg-br2cc\"" Oct 31 12:57:19.421661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.419948 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq"] Oct 31 12:57:19.450235 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450194 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450422 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450289 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450422 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450368 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450422 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450383 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450422 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450413 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450431 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54ps\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-kube-api-access-p54ps\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450457 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450478 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.450704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.450549 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552067 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552112 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552183 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552268 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552399 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552346 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552412 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552546 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552452 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552546 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552509 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552546 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552536 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54ps\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-kube-api-access-p54ps\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552734 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552612 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.552734 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552707 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.553434 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.552842 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.553434 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.553127 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.553434 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.553149 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.554755 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.554734 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.555142 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.555122 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.564190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.564163 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.565008 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.564983 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54ps\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-kube-api-access-p54ps\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:19.715626 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:19.715593 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:20.024031 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.023994 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq"] Oct 31 12:57:20.024237 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:57:20.024204 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a25f2_5cdf_41e8_9600_2ee1a2f3c8a3.slice/crio-f592ee208b79d62c8ca04124578963feda7720614ed19b256dc74c89d54258ea WatchSource:0}: Error finding container f592ee208b79d62c8ca04124578963feda7720614ed19b256dc74c89d54258ea: Status 404 returned error can't find the container with id f592ee208b79d62c8ca04124578963feda7720614ed19b256dc74c89d54258ea Oct 31 12:57:20.027055 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.027024 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:20.027151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.027090 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:20.027151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.027121 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:20.231418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.231378 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" event={"ID":"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3","Type":"ContainerStarted","Data":"7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0"} Oct 31 12:57:20.231577 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.231479 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" event={"ID":"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3","Type":"ContainerStarted","Data":"f592ee208b79d62c8ca04124578963feda7720614ed19b256dc74c89d54258ea"} Oct 31 12:57:20.260460 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.260413 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podStartSLOduration=1.260382921 podStartE2EDuration="1.260382921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:20.257242229 +0000 UTC m=+1534.434960211" watchObservedRunningTime="2026-10-31 12:57:20.260382921 +0000 UTC m=+1534.438100905" Oct 31 12:57:20.497835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.497788 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn"] Oct 31 12:57:20.503361 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.503335 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.523103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.521065 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:57:20.523103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.521201 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:57:20.523103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.521292 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:57:20.523103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.521371 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:57:20.523103 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.522972 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn"] Oct 31 12:57:20.566858 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.566826 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.567045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.566869 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.567045 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.567019 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdts2\" (UniqueName: \"kubernetes.io/projected/fad219e8-9e0e-44fc-ae40-88a218b8f760-kube-api-access-kdts2\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.668182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.668119 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.668375 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.668264 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.668442 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.668422 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdts2\" (UniqueName: \"kubernetes.io/projected/fad219e8-9e0e-44fc-ae40-88a218b8f760-kube-api-access-kdts2\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.668956 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.668922 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.671705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.671676 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.682447 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.682392 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdts2\" (UniqueName: \"kubernetes.io/projected/fad219e8-9e0e-44fc-ae40-88a218b8f760-kube-api-access-kdts2\") pod \"maas-api-revoke-sub-f1f22068-qtnfn\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:20.716949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.716850 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:20.826707 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:20.825940 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:21.314583 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.314266 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn"] Oct 31 12:57:21.629676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.629573 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff"] Oct 31 12:57:21.644666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.644472 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff"] Oct 31 12:57:21.644666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.644580 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.649688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.649355 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c4167730\"/\"e2e-lifecycle-model-c4167730-kserve-self-signed-certs\"" Oct 31 12:57:21.649688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.649387 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c4167730\"/\"default-dockercfg-b5kzw\"" Oct 31 12:57:21.649688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.649454 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c4167730\"/\"kube-root-ca.crt\"" Oct 31 12:57:21.649688 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.649484 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c4167730\"/\"openshift-service-ca.crt\"" Oct 31 12:57:21.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.730702 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 12:57:21.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.730775 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 12:57:21.836659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.831589 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tmp-dir\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.836659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.831701 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-home\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.836659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.831728 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tls-certs\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.836659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.831777 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-dshm\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.836659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.831836 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898bz\" (UniqueName: \"kubernetes.io/projected/15589bf4-60b5-4b6f-ae3f-114cba7703ef-kube-api-access-898bz\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.836659 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.832025 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-model-cache\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.955709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.955079 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-model-cache\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.955709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.952621 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-model-cache\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.955709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.955433 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tmp-dir\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.955709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.955500 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-home\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.955709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.955526 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tls-certs\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.955709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.955569 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-dshm\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.955709 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.955648 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898bz\" (UniqueName: \"kubernetes.io/projected/15589bf4-60b5-4b6f-ae3f-114cba7703ef-kube-api-access-898bz\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.956697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.956248 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tmp-dir\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:21.956697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:21.956468 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-home\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:22.031321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.026218 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tls-certs\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:22.031321 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.026589 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-dshm\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:22.050986 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.043720 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-898bz\" (UniqueName: \"kubernetes.io/projected/15589bf4-60b5-4b6f-ae3f-114cba7703ef-kube-api-access-898bz\") pod \"e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:22.251473 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.251416 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" event={"ID":"fad219e8-9e0e-44fc-ae40-88a218b8f760","Type":"ContainerStarted","Data":"d04a6adabf5fc7a2aef0eebc94e2613af6af7e450f68cf2acf98682a188d64c2"} Oct 31 12:57:22.251704 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.251482 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" event={"ID":"fad219e8-9e0e-44fc-ae40-88a218b8f760","Type":"ContainerStarted","Data":"182a49ce6a175bd1605cdf4afe2d1bd87212e36eeab65868b0431f7d7c4bbe1d"} Oct 31 12:57:22.260746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.260278 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:22.299658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.299477 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" podStartSLOduration=2.299454286 podStartE2EDuration="2.299454286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:22.274897728 +0000 UTC m=+1536.452615714" watchObservedRunningTime="2026-10-31 12:57:22.299454286 +0000 UTC m=+1536.477172271" Oct 31 12:57:22.719009 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.718976 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 12:57:22.719489 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.719027 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 12:57:22.909706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:22.909051 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff"] Oct 31 12:57:23.370661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:23.369435 5074 generic.go:356] "Generic (PLEG): container finished" podID="fad219e8-9e0e-44fc-ae40-88a218b8f760" containerID="d04a6adabf5fc7a2aef0eebc94e2613af6af7e450f68cf2acf98682a188d64c2" exitCode=0 Oct 31 12:57:23.370661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:23.369677 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" event={"ID":"fad219e8-9e0e-44fc-ae40-88a218b8f760","Type":"ContainerDied","Data":"d04a6adabf5fc7a2aef0eebc94e2613af6af7e450f68cf2acf98682a188d64c2"} Oct 31 12:57:23.448116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:23.448073 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" event={"ID":"15589bf4-60b5-4b6f-ae3f-114cba7703ef","Type":"ContainerStarted","Data":"8011b9230daa7a6d9e3833dad82ea365cf3bdefc4c21e657cc280d3059e69142"} Oct 31 12:57:23.723654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:23.718040 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 12:57:23.723654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:23.718106 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 12:57:24.717668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:24.717122 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 12:57:24.717668 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:24.717204 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 12:57:24.775820 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:24.775457 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:24.854254 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:24.851731 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:24.894778 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:24.894715 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 12:57:24.933473 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:24.933430 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.218855 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.218812 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 31 12:57:25.219518 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.219472 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 12:57:25.220791 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.220765 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.515232 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.515182 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 12:57:25.623684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.568607 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.623684 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.569041 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.638424 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.569071 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 12:57:25.698701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.694011 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" event={"ID":"15589bf4-60b5-4b6f-ae3f-114cba7703ef","Type":"ContainerStarted","Data":"0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318"} Oct 31 12:57:25.698701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.697695 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" event={"ID":"fad219e8-9e0e-44fc-ae40-88a218b8f760","Type":"ContainerDied","Data":"182a49ce6a175bd1605cdf4afe2d1bd87212e36eeab65868b0431f7d7c4bbe1d"} Oct 31 12:57:25.698701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.697757 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="182a49ce6a175bd1605cdf4afe2d1bd87212e36eeab65868b0431f7d7c4bbe1d" Oct 31 12:57:25.751774 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.751741 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:25.773660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.772650 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 12:57:25.773660 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.772730 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 12:57:25.779293 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.776831 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.779293 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.778324 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.780652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.779926 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:57:25.863656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.859548 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.875000 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.874955 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:57:25.917411 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.910719 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 12:57:25.938382 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.937230 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:25.989387 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.964878 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:57:25.989387 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:25.964989 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:57:26.260711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.156208 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" podStartSLOduration=5.15618799 podStartE2EDuration="5.15618799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:25.72991212 +0000 UTC m=+1539.907630105" watchObservedRunningTime="2026-10-31 12:57:26.15618799 +0000 UTC m=+1540.333905981" Oct 31 12:57:26.260711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.224553 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-openshift-ca\") pod \"fad219e8-9e0e-44fc-ae40-88a218b8f760\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " Oct 31 12:57:26.260711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.224712 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad219e8-9e0e-44fc-ae40-88a218b8f760-kube-api-access-kdts2\" (UniqueName: \"kubernetes.io/projected/fad219e8-9e0e-44fc-ae40-88a218b8f760-kube-api-access-kdts2\") pod \"fad219e8-9e0e-44fc-ae40-88a218b8f760\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " Oct 31 12:57:26.260711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.224855 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-xks-ca\") pod \"fad219e8-9e0e-44fc-ae40-88a218b8f760\" (UID: \"fad219e8-9e0e-44fc-ae40-88a218b8f760\") " Oct 31 12:57:26.260711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.226975 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-openshift-ca" pod "fad219e8-9e0e-44fc-ae40-88a218b8f760" (UID: "fad219e8-9e0e-44fc-ae40-88a218b8f760"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:57:26.281820 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.281777 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:26.283309 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.283276 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 31 12:57:26.327746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.327697 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:26.483938 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.483895 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad219e8-9e0e-44fc-ae40-88a218b8f760-kube-api-access-kdts2" pod "fad219e8-9e0e-44fc-ae40-88a218b8f760" (UID: "fad219e8-9e0e-44fc-ae40-88a218b8f760"). InnerVolumeSpecName "kube-api-access-kdts2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:57:26.518691 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.518577 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-xks-ca" pod "fad219e8-9e0e-44fc-ae40-88a218b8f760" (UID: "fad219e8-9e0e-44fc-ae40-88a218b8f760"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:57:26.577454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.577409 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 12:57:26.593293 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.577988 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:26.595171 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.595147 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:57:26.595492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.595473 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:26.597710 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.581227 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdts2\" (UniqueName: \"kubernetes.io/projected/fad219e8-9e0e-44fc-ae40-88a218b8f760-kube-api-access-kdts2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:26.597859 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.597843 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad219e8-9e0e-44fc-ae40-88a218b8f760-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:26.615669 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.615610 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 12:57:26.616481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.616457 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:57:26.751812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.749286 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 12:57:26.751812 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.749344 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 12:57:26.771663 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.771566 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn" Oct 31 12:57:26.797804 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.797567 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:26.798397 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.797596 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 12:57:26.817588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.817556 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:57:26.869656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.869106 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:26.905498 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:57:26.905446 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:57:26.921858 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.921812 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 12:57:26.938427 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:26.938387 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:27.010336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.001729 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:27.085836 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.085756 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:27.169654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.155884 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:57:27.169654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.155952 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:57:27.169654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.157901 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 12:57:27.238944 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.238909 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:57:27.256010 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.254738 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 12:57:27.258971 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.258942 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 12:57:27.260171 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.260146 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 12:57:27.272872 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.272840 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 12:57:27.346076 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:57:27.345999 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad219e8_9e0e_44fc_ae40_88a218b8f760.slice\": RecentStats: unable to find data in memory cache]" Oct 31 12:57:27.717806 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.717774 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 12:57:27.717960 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:27.717831 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 12:57:28.202430 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:28.202345 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:57:28.204428 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:28.204401 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:57:29.760655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:29.759662 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 12:57:29.760655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:29.759753 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:30.038791 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:30.038699 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:30.039109 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:30.039081 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:30.039781 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:30.039758 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 12:57:32.260552 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:32.260508 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:32.273570 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:32.273545 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:32.885149 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:32.885105 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:57:33.589385 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.589349 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x"] Oct 31 12:57:33.590216 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.590199 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fad219e8-9e0e-44fc-ae40-88a218b8f760" containerName="revoke-keys" Oct 31 12:57:33.590273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.590219 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad219e8-9e0e-44fc-ae40-88a218b8f760" containerName="revoke-keys" Oct 31 12:57:33.590389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.590378 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad219e8-9e0e-44fc-ae40-88a218b8f760" containerName="revoke-keys" Oct 31 12:57:33.595932 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.595903 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.600647 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.600606 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x"] Oct 31 12:57:33.601220 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.601197 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:57:33.601345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.601238 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:57:33.683732 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.683697 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6xl\" (UniqueName: \"kubernetes.io/projected/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-kube-api-access-sx6xl\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.683993 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.683848 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.684053 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.683986 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.686237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.686213 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:57:33.686358 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.686250 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:57:33.785403 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.785369 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6xl\" (UniqueName: \"kubernetes.io/projected/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-kube-api-access-sx6xl\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.785565 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.785443 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.785565 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.785500 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.786190 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.786167 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.787931 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.787905 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.798177 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.798152 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6xl\" (UniqueName: \"kubernetes.io/projected/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-kube-api-access-sx6xl\") pod \"maas-api-revoke-sub-bfdb78e7-f657x\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:33.919926 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:33.919841 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:34.243665 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:34.243607 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x"] Oct 31 12:57:34.244176 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:57:34.244151 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0371edb_ad02_4fe8_b6ac_a921cf0e1720.slice/crio-8a3b84611a14f7351745c1cc91d718d20c67c861ccc7b2b939f024a4681854cd WatchSource:0}: Error finding container 8a3b84611a14f7351745c1cc91d718d20c67c861ccc7b2b939f024a4681854cd: Status 404 returned error can't find the container with id 8a3b84611a14f7351745c1cc91d718d20c67c861ccc7b2b939f024a4681854cd Oct 31 12:57:34.900081 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:34.900046 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" event={"ID":"d0371edb-ad02-4fe8-b6ac-a921cf0e1720","Type":"ContainerStarted","Data":"8a3b84611a14f7351745c1cc91d718d20c67c861ccc7b2b939f024a4681854cd"} Oct 31 12:57:35.751242 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.751199 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw"] Oct 31 12:57:35.757351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.757327 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.771690 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.771662 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw"] Oct 31 12:57:35.774562 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.774538 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-73503b-openshift-default-dockercfg-zxk9v\"" Oct 31 12:57:35.813154 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813113 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-credential-socket\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813324 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813163 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-certs\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813324 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813279 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-data\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813324 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813313 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-podinfo\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813342 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzwd\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-kube-api-access-knzwd\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813488 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813390 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-token\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813584 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813566 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-envoy\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813701 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813682 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78e6005-f207-488d-a0c4-3e783888b3b2-istiod-ca-cert\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.813775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.813729 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-socket\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.926661 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.925590 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-token\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.927162 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.926250 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-envoy\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.927857 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.927832 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78e6005-f207-488d-a0c4-3e783888b3b2-istiod-ca-cert\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.929996 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.929973 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78e6005-f207-488d-a0c4-3e783888b3b2-istiod-ca-cert\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.939350 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.939325 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-socket\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.939508 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.939380 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-credential-socket\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.939508 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.939407 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-certs\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.939628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.939549 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-data\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.939628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.939589 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-podinfo\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.939628 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.939620 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzwd\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-kube-api-access-knzwd\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.950280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.940325 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-socket\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.950280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.940567 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-credential-socket\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.950280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.940809 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-certs\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.950280 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.941018 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-data\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:35.994410 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:35.993732 5074 generic.go:356] "Generic (PLEG): container finished" podID="d0371edb-ad02-4fe8-b6ac-a921cf0e1720" containerID="05b24aca2eb2a10a953ee4e1ae0dfff96b09611f4f91567bdc624c9c2a47bc99" exitCode=0 Oct 31 12:57:36.009719 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.001203 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-podinfo\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:36.049901 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.049867 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-token\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:36.082887 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.082675 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzwd\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-kube-api-access-knzwd\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:36.101168 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.101130 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-envoy\") pod \"e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:36.104839 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.103580 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" event={"ID":"d0371edb-ad02-4fe8-b6ac-a921cf0e1720","Type":"ContainerDied","Data":"05b24aca2eb2a10a953ee4e1ae0dfff96b09611f4f91567bdc624c9c2a47bc99"} Oct 31 12:57:36.388944 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.388848 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:36.696730 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.696696 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw"] Oct 31 12:57:36.697776 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:57:36.697745 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78e6005_f207_488d_a0c4_3e783888b3b2.slice/crio-c2f9703a1b93e248be4eebddbd19cab25e2bd0c21b5d8ba668cc14cc2fdfaa0e WatchSource:0}: Error finding container c2f9703a1b93e248be4eebddbd19cab25e2bd0c21b5d8ba668cc14cc2fdfaa0e: Status 404 returned error can't find the container with id c2f9703a1b93e248be4eebddbd19cab25e2bd0c21b5d8ba668cc14cc2fdfaa0e Oct 31 12:57:36.701102 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.701067 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:36.701208 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.701152 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:36.701208 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:36.701195 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:36.756237 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:57:36.753683 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78e6005_f207_488d_a0c4_3e783888b3b2.slice/crio-conmon-3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b.scope\": RecentStats: unable to find data in memory cache]" Oct 31 12:57:37.004847 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.004799 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" event={"ID":"f78e6005-f207-488d-a0c4-3e783888b3b2","Type":"ContainerStarted","Data":"3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b"} Oct 31 12:57:37.005329 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.004893 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" event={"ID":"f78e6005-f207-488d-a0c4-3e783888b3b2","Type":"ContainerStarted","Data":"c2f9703a1b93e248be4eebddbd19cab25e2bd0c21b5d8ba668cc14cc2fdfaa0e"} Oct 31 12:57:37.041674 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.039663 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podStartSLOduration=2.039627296 podStartE2EDuration="2.039627296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:37.029000497 +0000 UTC m=+1551.206718492" watchObservedRunningTime="2026-10-31 12:57:37.039627296 +0000 UTC m=+1551.217345280" Oct 31 12:57:37.264226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.264165 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:37.389493 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.389457 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:37.396626 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.396595 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:37.404876 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.404849 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-xks-ca\") pod \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " Oct 31 12:57:37.405003 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.404942 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-openshift-ca\") pod \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " Oct 31 12:57:37.405059 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.405005 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-kube-api-access-sx6xl\" (UniqueName: \"kubernetes.io/projected/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-kube-api-access-sx6xl\") pod \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\" (UID: \"d0371edb-ad02-4fe8-b6ac-a921cf0e1720\") " Oct 31 12:57:37.405380 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.405355 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-openshift-ca" pod "d0371edb-ad02-4fe8-b6ac-a921cf0e1720" (UID: "d0371edb-ad02-4fe8-b6ac-a921cf0e1720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:57:37.405541 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.405524 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:37.407566 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.407542 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-xks-ca" pod "d0371edb-ad02-4fe8-b6ac-a921cf0e1720" (UID: "d0371edb-ad02-4fe8-b6ac-a921cf0e1720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:57:37.407696 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.407595 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-kube-api-access-sx6xl" pod "d0371edb-ad02-4fe8-b6ac-a921cf0e1720" (UID: "d0371edb-ad02-4fe8-b6ac-a921cf0e1720"). InnerVolumeSpecName "kube-api-access-sx6xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:57:37.506966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.506935 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx6xl\" (UniqueName: \"kubernetes.io/projected/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-kube-api-access-sx6xl\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:37.506966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:37.506964 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0371edb-ad02-4fe8-b6ac-a921cf0e1720-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:38.015334 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:38.015299 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" Oct 31 12:57:38.015334 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:38.015326 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x" event={"ID":"d0371edb-ad02-4fe8-b6ac-a921cf0e1720","Type":"ContainerDied","Data":"8a3b84611a14f7351745c1cc91d718d20c67c861ccc7b2b939f024a4681854cd"} Oct 31 12:57:38.015924 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:38.015387 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a3b84611a14f7351745c1cc91d718d20c67c861ccc7b2b939f024a4681854cd" Oct 31 12:57:38.015924 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:38.015802 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:38.016796 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:38.016775 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:57:43.380680 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.380622 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:57:43.946320 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.946284 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs"] Oct 31 12:57:43.946977 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.946955 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d0371edb-ad02-4fe8-b6ac-a921cf0e1720" containerName="revoke-keys" Oct 31 12:57:43.947097 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.946978 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0371edb-ad02-4fe8-b6ac-a921cf0e1720" containerName="revoke-keys" Oct 31 12:57:43.947206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.947191 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0371edb-ad02-4fe8-b6ac-a921cf0e1720" containerName="revoke-keys" Oct 31 12:57:43.950953 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.950934 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:43.980196 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.980161 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:57:43.980331 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.980312 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:57:43.980421 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.980399 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:57:43.981792 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.981773 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:57:43.987069 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:43.987047 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs"] Oct 31 12:57:44.016076 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.016043 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.016251 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.016226 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.016312 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.016295 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njg5m\" (UniqueName: \"kubernetes.io/projected/029134c2-ae8e-45b9-8ffb-a800c223758f-kube-api-access-njg5m\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.118165 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.118127 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.118318 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.118257 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.118318 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.118312 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njg5m\" (UniqueName: \"kubernetes.io/projected/029134c2-ae8e-45b9-8ffb-a800c223758f-kube-api-access-njg5m\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.119034 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.119003 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.120776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.120749 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.139655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.139238 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njg5m\" (UniqueName: \"kubernetes.io/projected/029134c2-ae8e-45b9-8ffb-a800c223758f-kube-api-access-njg5m\") pod \"maas-api-revoke-sub-264abcf4-wlnbs\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:44.267877 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:44.267840 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:45.106315 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.106284 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv"] Oct 31 12:57:45.237658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.228301 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv"] Oct 31 12:57:45.237658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.228357 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs"] Oct 31 12:57:45.237658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.228455 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280525 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac29fcc-c69a-45b2-8189-70d35d8f4765-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280596 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280626 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625zw\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-kube-api-access-625zw\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280678 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280729 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280750 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280770 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280873 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.280934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.280896 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.434679 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac29fcc-c69a-45b2-8189-70d35d8f4765-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.434751 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.434787 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625zw\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-kube-api-access-625zw\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.434824 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.434883 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.434908 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.434937 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.437305 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-credential-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.447657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.437570 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-data\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.451650 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.449750 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.451650 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.449788 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.451650 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.450662 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-socket\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.461865 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.460295 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac29fcc-c69a-45b2-8189-70d35d8f4765-istiod-ca-cert\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.461865 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.460705 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-certs\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.579656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.569258 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-podinfo\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.579656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.571195 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625zw\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-kube-api-access-625zw\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.579656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.575862 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-token\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.580234 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.580210 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-envoy\") pod \"e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.657703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.654441 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:45.905462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:45.903696 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.242398 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.162432 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 31 12:57:46.276662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.199247 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.276662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.199286 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 12:57:46.276662 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.253715 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.304307 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.302720 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.432159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.430481 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 12:57:46.510191 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.509415 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:46.510191 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.509503 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:46.510191 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.509539 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:46.541213 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.539931 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" event={"ID":"029134c2-ae8e-45b9-8ffb-a800c223758f","Type":"ContainerStarted","Data":"765d2a683ba1853a31211c2cdc14f1d019d15fa6561b2792258c6742ea1b9675"} Oct 31 12:57:46.551914 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.551745 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.573648 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.570031 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv"] Oct 31 12:57:46.890654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.883742 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" podUID="15589bf4-60b5-4b6f-ae3f-114cba7703ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:46.997003 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:46.992214 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 12:57:47.084517 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.083778 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.257795 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.257746 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 12:57:47.262203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.258854 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.262203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.259239 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:57:47.262203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.259829 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:47.262203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.260221 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 12:57:47.262203 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.260517 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:57:47.286536 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.285559 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:57:47.811652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.810987 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" event={"ID":"cac29fcc-c69a-45b2-8189-70d35d8f4765","Type":"ContainerStarted","Data":"c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a"} Oct 31 12:57:47.811652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.811098 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" event={"ID":"cac29fcc-c69a-45b2-8189-70d35d8f4765","Type":"ContainerStarted","Data":"7aaf2763a286894dd4544b8502cc64d40c1ca0a57830e68cc39c356412b16211"} Oct 31 12:57:47.824622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.814520 5074 generic.go:356] "Generic (PLEG): container finished" podID="029134c2-ae8e-45b9-8ffb-a800c223758f" containerID="d4f683445e5d9b256a03d17b00dcaf4626ba60558008e93fbd66b0cdaf19d977" exitCode=0 Oct 31 12:57:47.824622 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.814704 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" event={"ID":"029134c2-ae8e-45b9-8ffb-a800c223758f","Type":"ContainerDied","Data":"d4f683445e5d9b256a03d17b00dcaf4626ba60558008e93fbd66b0cdaf19d977"} Oct 31 12:57:47.843002 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.842966 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 12:57:47.846758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.845283 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 12:57:47.974216 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:47.972306 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podStartSLOduration=3.972286626 podStartE2EDuration="3.972286626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:47.894817317 +0000 UTC m=+1562.072535305" watchObservedRunningTime="2026-10-31 12:57:47.972286626 +0000 UTC m=+1562.150004610" Oct 31 12:57:48.183332 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:48.152872 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 12:57:48.666652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:48.658193 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:48.666652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:48.659732 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 12:57:48.666652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:48.659780 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 12:57:48.899775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:48.899732 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 12:57:48.927743 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:48.925294 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:49.019389 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.018740 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:49.158769 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.158719 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:49.525654 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.524962 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 12:57:49.613792 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.613720 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq"] Oct 31 12:57:49.666044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.659622 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.666044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.661984 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 12:57:49.666044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.662036 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 12:57:49.700131 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.697224 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq"] Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.722603 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.722665 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.722728 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsl2r\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-kube-api-access-qsl2r\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.722785 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.722816 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.722838 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.722888 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.723059 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.731345 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.723141 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22c34a8-e6b9-4e0a-bd18-177413888baa-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824491 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22c34a8-e6b9-4e0a-bd18-177413888baa-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824598 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824623 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824690 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsl2r\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-kube-api-access-qsl2r\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824739 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824768 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824794 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824840 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.825534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.824995 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.826354 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.826337 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:49.852159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.843268 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22c34a8-e6b9-4e0a-bd18-177413888baa-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.852159 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.844407 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.852745 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.852719 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.900727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.884179 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.900727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.884865 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.900727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.885481 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsl2r\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-kube-api-access-qsl2r\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.900727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.899873 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" Oct 31 12:57:49.900727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.900027 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs" event={"ID":"029134c2-ae8e-45b9-8ffb-a800c223758f","Type":"ContainerDied","Data":"765d2a683ba1853a31211c2cdc14f1d019d15fa6561b2792258c6742ea1b9675"} Oct 31 12:57:49.900727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.900068 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="765d2a683ba1853a31211c2cdc14f1d019d15fa6561b2792258c6742ea1b9675" Oct 31 12:57:49.900727 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.900482 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.901717 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.901324 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:49.936851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.929239 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-openshift-ca\") pod \"029134c2-ae8e-45b9-8ffb-a800c223758f\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " Oct 31 12:57:49.937188 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.937159 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-xks-ca\") pod \"029134c2-ae8e-45b9-8ffb-a800c223758f\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " Oct 31 12:57:49.937573 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.937551 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029134c2-ae8e-45b9-8ffb-a800c223758f-kube-api-access-njg5m\" (UniqueName: \"kubernetes.io/projected/029134c2-ae8e-45b9-8ffb-a800c223758f-kube-api-access-njg5m\") pod \"029134c2-ae8e-45b9-8ffb-a800c223758f\" (UID: \"029134c2-ae8e-45b9-8ffb-a800c223758f\") " Oct 31 12:57:49.996750 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.979212 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:49.996750 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:49.988171 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029134c2-ae8e-45b9-8ffb-a800c223758f-kube-api-access-njg5m" pod "029134c2-ae8e-45b9-8ffb-a800c223758f" (UID: "029134c2-ae8e-45b9-8ffb-a800c223758f"). InnerVolumeSpecName "kube-api-access-njg5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:57:50.115896 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.104421 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-xks-ca" pod "029134c2-ae8e-45b9-8ffb-a800c223758f" (UID: "029134c2-ae8e-45b9-8ffb-a800c223758f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:57:50.115896 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.105066 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:50.116323 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.116291 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:50.146135 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.146094 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-openshift-ca" pod "029134c2-ae8e-45b9-8ffb-a800c223758f" (UID: "029134c2-ae8e-45b9-8ffb-a800c223758f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:57:50.156044 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.155977 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njg5m\" (UniqueName: \"kubernetes.io/projected/029134c2-ae8e-45b9-8ffb-a800c223758f-kube-api-access-njg5m\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:50.156296 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.156274 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:50.156446 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.156431 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029134c2-ae8e-45b9-8ffb-a800c223758f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:50.158251 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.158208 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 12:57:50.604080 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.602856 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:50.604080 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.602956 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:50.604080 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.602999 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:57:50.624567 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.624519 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq"] Oct 31 12:57:50.656117 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.655721 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 12:57:50.656117 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.655785 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 12:57:50.929425 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.929334 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" event={"ID":"a22c34a8-e6b9-4e0a-bd18-177413888baa","Type":"ContainerStarted","Data":"f06a7411202d801df9503d2529b194a79d71e37e7338ce0366c053052cbfb972"} Oct 31 12:57:50.931312 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.931286 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" event={"ID":"a22c34a8-e6b9-4e0a-bd18-177413888baa","Type":"ContainerStarted","Data":"8bc43761bc09cd98293ea30387b85b265614c4ea294088d2f5166047fff60bcb"} Oct 31 12:57:50.958444 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:50.958396 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podStartSLOduration=1.958382155 podStartE2EDuration="1.958382155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:50.953842552 +0000 UTC m=+1565.131560537" watchObservedRunningTime="2026-10-31 12:57:50.958382155 +0000 UTC m=+1565.136100139" Oct 31 12:57:51.150976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:51.150936 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:51.152338 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:51.152312 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 12:57:51.152444 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:51.152365 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 12:57:51.655414 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:51.655364 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 12:57:51.655589 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:51.655429 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 12:57:52.152399 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:52.152359 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 12:57:52.152833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:52.152425 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 12:57:52.655162 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:52.655122 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 12:57:52.655353 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:52.655187 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 12:57:53.152519 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:53.152477 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 12:57:53.152977 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:53.152560 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 12:57:53.655148 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:53.655113 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 12:57:53.655316 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:53.655173 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 12:57:54.151038 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:54.151002 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 12:57:54.151255 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:54.151055 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 12:57:54.655236 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:54.655178 5074 patch_prober.go:29] interesting pod/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 12:57:54.655829 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:54.655258 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 12:57:55.151574 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.151533 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 12:57:55.151768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.151601 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 12:57:55.214119 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.214081 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj"] Oct 31 12:57:55.214714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.214697 5074 cpu_manager.go:414] "containerMap: removing container" podUID="029134c2-ae8e-45b9-8ffb-a800c223758f" containerName="revoke-keys" Oct 31 12:57:55.214714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.214713 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029134c2-ae8e-45b9-8ffb-a800c223758f" containerName="revoke-keys" Oct 31 12:57:55.214875 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.214860 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="029134c2-ae8e-45b9-8ffb-a800c223758f" containerName="revoke-keys" Oct 31 12:57:55.219655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.219619 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.227252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.227228 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:57:55.227940 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.227917 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:57:55.228182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.228165 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:57:55.228182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.228176 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:57:55.228800 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.228769 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj"] Oct 31 12:57:55.332445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.332379 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.332658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.332546 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.332658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.332593 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwwj\" (UniqueName: \"kubernetes.io/projected/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-kube-api-access-tzwwj\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.434882 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.434791 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.435057 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.434903 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.435057 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.434941 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwwj\" (UniqueName: \"kubernetes.io/projected/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-kube-api-access-tzwwj\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.435875 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.435851 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.438034 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.438008 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.447982 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.447954 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwwj\" (UniqueName: \"kubernetes.io/projected/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-kube-api-access-tzwwj\") pod \"maas-api-revoke-sub-dc55c842-h88jj\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.543436 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.543391 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:55.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.926065 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj"] Oct 31 12:57:55.987967 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.987937 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:55.989706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:55.989682 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:56.013100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:56.013041 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 12:57:56.017871 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:57:56.017818 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:57:56.047125 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:56.047090 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" event={"ID":"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5","Type":"ContainerStarted","Data":"4e7110c2c51cfc8703b14f7e34a2cad2ba54deb7083b813f36355aa762647ace"} Oct 31 12:57:56.151866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:56.151765 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 12:57:56.151866 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:56.151813 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 12:57:57.056604 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:57.056566 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" event={"ID":"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5","Type":"ContainerStarted","Data":"4083f2c390d15142737f5a8d637333eec13caf3be961c9db9bb25313819fe599"} Oct 31 12:57:57.081258 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:57.081208 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" podStartSLOduration=2.081194418 podStartE2EDuration="2.081194418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:57:57.07877611 +0000 UTC m=+1571.256494094" watchObservedRunningTime="2026-10-31 12:57:57.081194418 +0000 UTC m=+1571.258912400" Oct 31 12:57:57.151478 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:57.151447 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 12:57:57.151691 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:57.151502 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 12:57:58.068513 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:58.068472 5074 generic.go:356] "Generic (PLEG): container finished" podID="c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" containerID="4083f2c390d15142737f5a8d637333eec13caf3be961c9db9bb25313819fe599" exitCode=0 Oct 31 12:57:58.068957 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:58.068560 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" event={"ID":"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5","Type":"ContainerDied","Data":"4083f2c390d15142737f5a8d637333eec13caf3be961c9db9bb25313819fe599"} Oct 31 12:57:58.336018 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:58.335932 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:58.336492 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:58.336463 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:58.337789 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:58.337768 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 12:57:59.279179 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.279149 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:57:59.409473 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.409307 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-openshift-ca\") pod \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " Oct 31 12:57:59.409473 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.409369 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-kube-api-access-tzwwj\" (UniqueName: \"kubernetes.io/projected/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-kube-api-access-tzwwj\") pod \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " Oct 31 12:57:59.409775 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.409677 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-xks-ca\") pod \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\" (UID: \"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5\") " Oct 31 12:57:59.410619 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.410588 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-openshift-ca" pod "c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" (UID: "c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:57:59.412295 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.412272 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-xks-ca" pod "c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" (UID: "c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:57:59.412394 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.412272 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-kube-api-access-tzwwj" pod "c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" (UID: "c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5"). InnerVolumeSpecName "kube-api-access-tzwwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:57:59.511648 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.511612 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:59.511648 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.511652 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:57:59.511835 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:57:59.511663 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzwwj\" (UniqueName: \"kubernetes.io/projected/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5-kube-api-access-tzwwj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:00.087655 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:00.087614 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" Oct 31 12:58:00.088174 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:00.088152 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj" event={"ID":"c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5","Type":"ContainerDied","Data":"4e7110c2c51cfc8703b14f7e34a2cad2ba54deb7083b813f36355aa762647ace"} Oct 31 12:58:00.088315 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:00.088195 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7110c2c51cfc8703b14f7e34a2cad2ba54deb7083b813f36355aa762647ace" Oct 31 12:58:08.482133 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.482092 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw"] Oct 31 12:58:08.482806 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.482791 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" containerName="revoke-keys" Oct 31 12:58:08.482806 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.482806 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" containerName="revoke-keys" Oct 31 12:58:08.482918 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.482912 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" containerName="revoke-keys" Oct 31 12:58:08.486770 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.486754 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.493899 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.493366 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:58:08.493899 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.493443 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:58:08.493899 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.493529 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:58:08.493899 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.493528 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:58:08.497242 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.497009 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw"] Oct 31 12:58:08.540413 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.540383 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.540591 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.540477 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmhd\" (UniqueName: \"kubernetes.io/projected/aad001eb-4058-4495-abf0-20aefbc9dd21-kube-api-access-2xmhd\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.540591 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.540547 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.641901 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.641867 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmhd\" (UniqueName: \"kubernetes.io/projected/aad001eb-4058-4495-abf0-20aefbc9dd21-kube-api-access-2xmhd\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.642082 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.641911 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.642131 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.642089 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.642711 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.642684 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.644296 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.644276 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.654956 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.654934 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmhd\" (UniqueName: \"kubernetes.io/projected/aad001eb-4058-4495-abf0-20aefbc9dd21-kube-api-access-2xmhd\") pod \"maas-api-revoke-sub-fe4c330e-n69rw\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:08.805663 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:08.805542 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:09.111104 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:58:09.110855 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad001eb_4058_4495_abf0_20aefbc9dd21.slice/crio-25497dd31463ca04649af5b36b71e274117ee99b5f8ff82aafcd0205065c4106 WatchSource:0}: Error finding container 25497dd31463ca04649af5b36b71e274117ee99b5f8ff82aafcd0205065c4106: Status 404 returned error can't find the container with id 25497dd31463ca04649af5b36b71e274117ee99b5f8ff82aafcd0205065c4106 Oct 31 12:58:09.113578 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:09.113550 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw"] Oct 31 12:58:09.178121 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:09.178083 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" event={"ID":"aad001eb-4058-4495-abf0-20aefbc9dd21","Type":"ContainerStarted","Data":"25497dd31463ca04649af5b36b71e274117ee99b5f8ff82aafcd0205065c4106"} Oct 31 12:58:10.189310 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:10.189274 5074 generic.go:356] "Generic (PLEG): container finished" podID="aad001eb-4058-4495-abf0-20aefbc9dd21" containerID="de7f1c41d1ad2a9e873803fe6731277d014f037b46203dfae81e2ed91ffe0a65" exitCode=0 Oct 31 12:58:10.189942 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:10.189499 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" event={"ID":"aad001eb-4058-4495-abf0-20aefbc9dd21","Type":"ContainerDied","Data":"de7f1c41d1ad2a9e873803fe6731277d014f037b46203dfae81e2ed91ffe0a65"} Oct 31 12:58:11.478080 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.478049 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:11.592013 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.591983 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-openshift-ca\") pod \"aad001eb-4058-4495-abf0-20aefbc9dd21\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " Oct 31 12:58:11.592189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.592034 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-xks-ca\") pod \"aad001eb-4058-4495-abf0-20aefbc9dd21\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " Oct 31 12:58:11.592189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.592173 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad001eb-4058-4495-abf0-20aefbc9dd21-kube-api-access-2xmhd\" (UniqueName: \"kubernetes.io/projected/aad001eb-4058-4495-abf0-20aefbc9dd21-kube-api-access-2xmhd\") pod \"aad001eb-4058-4495-abf0-20aefbc9dd21\" (UID: \"aad001eb-4058-4495-abf0-20aefbc9dd21\") " Oct 31 12:58:11.592432 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.592403 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-openshift-ca" pod "aad001eb-4058-4495-abf0-20aefbc9dd21" (UID: "aad001eb-4058-4495-abf0-20aefbc9dd21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:11.592881 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.592857 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:11.594377 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.594344 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad001eb-4058-4495-abf0-20aefbc9dd21-kube-api-access-2xmhd" pod "aad001eb-4058-4495-abf0-20aefbc9dd21" (UID: "aad001eb-4058-4495-abf0-20aefbc9dd21"). InnerVolumeSpecName "kube-api-access-2xmhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:11.594476 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.594381 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-xks-ca" pod "aad001eb-4058-4495-abf0-20aefbc9dd21" (UID: "aad001eb-4058-4495-abf0-20aefbc9dd21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:11.694505 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.694470 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad001eb-4058-4495-abf0-20aefbc9dd21-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:11.694505 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:11.694507 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmhd\" (UniqueName: \"kubernetes.io/projected/aad001eb-4058-4495-abf0-20aefbc9dd21-kube-api-access-2xmhd\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:12.213695 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:12.213602 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" Oct 31 12:58:12.213695 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:12.213620 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw" event={"ID":"aad001eb-4058-4495-abf0-20aefbc9dd21","Type":"ContainerDied","Data":"25497dd31463ca04649af5b36b71e274117ee99b5f8ff82aafcd0205065c4106"} Oct 31 12:58:12.213908 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:12.213719 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25497dd31463ca04649af5b36b71e274117ee99b5f8ff82aafcd0205065c4106" Oct 31 12:58:18.141533 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.141497 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5"] Oct 31 12:58:18.148579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.146769 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl"] Oct 31 12:58:18.773509 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.773465 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs"] Oct 31 12:58:18.774400 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.774379 5074 cpu_manager.go:414] "containerMap: removing container" podUID="aad001eb-4058-4495-abf0-20aefbc9dd21" containerName="revoke-keys" Oct 31 12:58:18.774495 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.774402 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad001eb-4058-4495-abf0-20aefbc9dd21" containerName="revoke-keys" Oct 31 12:58:18.774617 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.774605 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad001eb-4058-4495-abf0-20aefbc9dd21" containerName="revoke-keys" Oct 31 12:58:18.780333 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.780308 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:18.790169 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.790143 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:58:18.794708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.791982 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:58:18.794867 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.792583 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:58:18.794947 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.792647 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:58:18.804084 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.804063 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs"] Oct 31 12:58:18.951014 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.950979 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d55j\" (UniqueName: \"kubernetes.io/projected/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-kube-api-access-6d55j\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:18.951199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.951176 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:18.951265 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:18.951236 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.052494 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.052352 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d55j\" (UniqueName: \"kubernetes.io/projected/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-kube-api-access-6d55j\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.052676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.052497 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.052676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.052532 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.053376 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.053349 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.055219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.055197 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.065973 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.065944 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d55j\" (UniqueName: \"kubernetes.io/projected/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-kube-api-access-6d55j\") pod \"maas-api-revoke-sub-77dff3e8-xmfrs\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.125169 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.125132 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:19.497834 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.496760 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs"] Oct 31 12:58:19.622920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.622875 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct"] Oct 31 12:58:19.633677 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.629127 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.663884 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.663839 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct"] Oct 31 12:58:19.671788 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.671750 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.672207 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.672166 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.672477 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.672458 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.672804 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.672783 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.673059 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.673040 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.673299 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.673281 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.673485 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.673468 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.673715 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.673697 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qc2\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-kube-api-access-n2qc2\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.673913 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.673893 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775369 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775478 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775514 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775572 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qc2\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-kube-api-access-n2qc2\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775618 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775608 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775685 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775727 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775756 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.775842 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.775791 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.777252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.776196 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-credential-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.777252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.776698 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-certs\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.777252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.776963 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-data\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.777252 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.777183 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-socket\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.777528 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.777409 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istiod-ca-cert\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.780759 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.780117 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-podinfo\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.781928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.781885 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-envoy\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.792014 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.791979 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-token\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.793049 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.793025 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qc2\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-kube-api-access-n2qc2\") pod \"e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:19.961445 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:19.961403 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:20.199932 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.199895 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11391-a96ade"] Oct 31 12:58:20.205578 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.205553 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:20.216197 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.216155 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mn5fx\"" Oct 31 12:58:20.220410 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.220384 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-a96ade"] Oct 31 12:58:20.287527 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.287492 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct"] Oct 31 12:58:20.293504 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.293478 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2lh\" (UniqueName: \"kubernetes.io/projected/2dc601c6-fd43-4a22-8abf-27356d92523d-kube-api-access-4z2lh\") pod \"test-curl-11391-a96ade\" (UID: \"2dc601c6-fd43-4a22-8abf-27356d92523d\") " pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:20.307676 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:58:20.307109 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cd1bd6_c4c3_4c73_9110_4f0b2f604c96.slice/crio-21c5d568339a8196e58d1efd824a921660e2d830068143756e39d9988e76808a WatchSource:0}: Error finding container 21c5d568339a8196e58d1efd824a921660e2d830068143756e39d9988e76808a: Status 404 returned error can't find the container with id 21c5d568339a8196e58d1efd824a921660e2d830068143756e39d9988e76808a Oct 31 12:58:20.311151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.310792 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:20.311151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.310888 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:20.311151 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.310932 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:20.316487 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.316451 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" event={"ID":"f9ff4cad-7f28-4999-8fdd-c3c63ab58870","Type":"ContainerStarted","Data":"07450ab8a6ebc7f719521df156343d452f2afd4253efe497c29d1b56d51e8890"} Oct 31 12:58:20.316601 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.316525 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" event={"ID":"f9ff4cad-7f28-4999-8fdd-c3c63ab58870","Type":"ContainerStarted","Data":"ffab4f167a2a8b9b51ce215f9b0275702f956bbb25b11a815f823d33d2f6cb7f"} Oct 31 12:58:20.355758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.355712 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" podStartSLOduration=2.355698779 podStartE2EDuration="2.355698779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:20.352766874 +0000 UTC m=+1594.530484876" watchObservedRunningTime="2026-10-31 12:58:20.355698779 +0000 UTC m=+1594.533416761" Oct 31 12:58:20.398705 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.395333 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2lh\" (UniqueName: \"kubernetes.io/projected/2dc601c6-fd43-4a22-8abf-27356d92523d-kube-api-access-4z2lh\") pod \"test-curl-11391-a96ade\" (UID: \"2dc601c6-fd43-4a22-8abf-27356d92523d\") " pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:20.419481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.419452 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2lh\" (UniqueName: \"kubernetes.io/projected/2dc601c6-fd43-4a22-8abf-27356d92523d-kube-api-access-4z2lh\") pod \"test-curl-11391-a96ade\" (UID: \"2dc601c6-fd43-4a22-8abf-27356d92523d\") " pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:20.532921 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.532852 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:20.842476 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.842446 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11391-a96ade"] Oct 31 12:58:20.843095 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:58:20.843066 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc601c6_fd43_4a22_8abf_27356d92523d.slice/crio-212b0ce33d141a0204a8a5515ac8ed28c16de4bc83ff22d37c2ee3239b354cec WatchSource:0}: Error finding container 212b0ce33d141a0204a8a5515ac8ed28c16de4bc83ff22d37c2ee3239b354cec: Status 404 returned error can't find the container with id 212b0ce33d141a0204a8a5515ac8ed28c16de4bc83ff22d37c2ee3239b354cec Oct 31 12:58:20.845380 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:20.845359 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 12:58:21.350652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.349996 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" event={"ID":"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96","Type":"ContainerStarted","Data":"2e0d19165e6f000c5f06712d21a5ecf07a3ee36eb2039cfc7d06e6ee6f094af2"} Oct 31 12:58:21.350652 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.350090 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" event={"ID":"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96","Type":"ContainerStarted","Data":"21c5d568339a8196e58d1efd824a921660e2d830068143756e39d9988e76808a"} Oct 31 12:58:21.357370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.354880 5074 generic.go:356] "Generic (PLEG): container finished" podID="f9ff4cad-7f28-4999-8fdd-c3c63ab58870" containerID="07450ab8a6ebc7f719521df156343d452f2afd4253efe497c29d1b56d51e8890" exitCode=0 Oct 31 12:58:21.357370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.355168 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" event={"ID":"f9ff4cad-7f28-4999-8fdd-c3c63ab58870","Type":"ContainerDied","Data":"07450ab8a6ebc7f719521df156343d452f2afd4253efe497c29d1b56d51e8890"} Oct 31 12:58:21.357370 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.357183 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-a96ade" event={"ID":"2dc601c6-fd43-4a22-8abf-27356d92523d","Type":"ContainerStarted","Data":"212b0ce33d141a0204a8a5515ac8ed28c16de4bc83ff22d37c2ee3239b354cec"} Oct 31 12:58:21.405706 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.405622 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podStartSLOduration=2.405602263 podStartE2EDuration="2.405602263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:21.398995383 +0000 UTC m=+1595.576713392" watchObservedRunningTime="2026-10-31 12:58:21.405602263 +0000 UTC m=+1595.583320248" Oct 31 12:58:21.961965 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.961926 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:21.963810 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.963776 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 12:58:21.963944 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:21.963847 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 12:58:22.661465 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.661406 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:22.735483 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.735427 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-kube-api-access-6d55j\" (UniqueName: \"kubernetes.io/projected/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-kube-api-access-6d55j\") pod \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " Oct 31 12:58:22.735685 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.735599 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-openshift-ca\") pod \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " Oct 31 12:58:22.735833 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.735816 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-xks-ca\") pod \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\" (UID: \"f9ff4cad-7f28-4999-8fdd-c3c63ab58870\") " Oct 31 12:58:22.736858 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.736832 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-openshift-ca" pod "f9ff4cad-7f28-4999-8fdd-c3c63ab58870" (UID: "f9ff4cad-7f28-4999-8fdd-c3c63ab58870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:22.738852 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.738826 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-xks-ca" pod "f9ff4cad-7f28-4999-8fdd-c3c63ab58870" (UID: "f9ff4cad-7f28-4999-8fdd-c3c63ab58870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:22.740881 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.740859 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-kube-api-access-6d55j" pod "f9ff4cad-7f28-4999-8fdd-c3c63ab58870" (UID: "f9ff4cad-7f28-4999-8fdd-c3c63ab58870"). InnerVolumeSpecName "kube-api-access-6d55j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:22.837266 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.837227 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:22.837266 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.837265 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d55j\" (UniqueName: \"kubernetes.io/projected/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-kube-api-access-6d55j\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:22.837503 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.837281 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ff4cad-7f28-4999-8fdd-c3c63ab58870-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:22.962357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.962284 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 12:58:22.962357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:22.962333 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 12:58:23.017357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.017319 5074 patch_prober.go:29] interesting pod/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 12:58:23.017538 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.017378 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 12:58:23.142465 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.142426 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" podUID="79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" containerName="payload-processing" containerID="cri-o://41139cf5a3be546c36e7d01332212bc3d16526a454a11bebfbd8da74394cc02c" gracePeriod=25 Oct 31 12:58:23.148015 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.147675 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" podUID="53634ad4-669e-4e07-96be-a14aa9c23543" containerName="payload-pre-processing" containerID="cri-o://ae00cad027a5cfee827e3c46f87de1df545251acc4dc684c817d2a3c4253e42e" gracePeriod=25 Oct 31 12:58:23.428702 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.428665 5074 generic.go:356] "Generic (PLEG): container finished" podID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerID="3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b" exitCode=0 Oct 31 12:58:23.429035 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.429011 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" event={"ID":"f78e6005-f207-488d-a0c4-3e783888b3b2","Type":"ContainerDied","Data":"3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b"} Oct 31 12:58:23.429722 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.429702 5074 scope.go:122] "RemoveContainer" containerID="3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b" Oct 31 12:58:23.432009 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.431979 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:23.432197 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.432178 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:23.432324 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.432309 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:58:23.434943 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.434919 5074 generic.go:356] "Generic (PLEG): container finished" podID="79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" containerID="41139cf5a3be546c36e7d01332212bc3d16526a454a11bebfbd8da74394cc02c" exitCode=0 Oct 31 12:58:23.435055 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.435003 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" event={"ID":"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2","Type":"ContainerDied","Data":"41139cf5a3be546c36e7d01332212bc3d16526a454a11bebfbd8da74394cc02c"} Oct 31 12:58:23.439172 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.439156 5074 generic.go:356] "Generic (PLEG): container finished" podID="53634ad4-669e-4e07-96be-a14aa9c23543" containerID="ae00cad027a5cfee827e3c46f87de1df545251acc4dc684c817d2a3c4253e42e" exitCode=0 Oct 31 12:58:23.439423 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.439411 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" event={"ID":"53634ad4-669e-4e07-96be-a14aa9c23543","Type":"ContainerDied","Data":"ae00cad027a5cfee827e3c46f87de1df545251acc4dc684c817d2a3c4253e42e"} Oct 31 12:58:23.446552 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.446528 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" Oct 31 12:58:23.446829 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.446808 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs" event={"ID":"f9ff4cad-7f28-4999-8fdd-c3c63ab58870","Type":"ContainerDied","Data":"ffab4f167a2a8b9b51ce215f9b0275702f956bbb25b11a815f823d33d2f6cb7f"} Oct 31 12:58:23.446876 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.446855 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffab4f167a2a8b9b51ce215f9b0275702f956bbb25b11a815f823d33d2f6cb7f" Oct 31 12:58:23.515089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.514269 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:58:23.549742 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.549267 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-kube-api-access-gx99v\" (UniqueName: \"kubernetes.io/projected/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-kube-api-access-gx99v\") pod \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " Oct 31 12:58:23.549742 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.549479 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-plugins-config-volume\") pod \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\" (UID: \"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2\") " Oct 31 12:58:23.551786 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.551526 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-plugins-config-volume" pod "79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" (UID: "79246ba3-75f6-4afa-9ac0-9448e5bdf6f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:23.562780 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.562732 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-kube-api-access-gx99v" pod "79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" (UID: "79246ba3-75f6-4afa-9ac0-9448e5bdf6f2"). InnerVolumeSpecName "kube-api-access-gx99v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:23.652588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.652374 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:23.652588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.652404 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx99v\" (UniqueName: \"kubernetes.io/projected/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2-kube-api-access-gx99v\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:23.705429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.705053 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:58:23.754723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.754104 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53634ad4-669e-4e07-96be-a14aa9c23543-kube-api-access-gc6f7\" (UniqueName: \"kubernetes.io/projected/53634ad4-669e-4e07-96be-a14aa9c23543-kube-api-access-gc6f7\") pod \"53634ad4-669e-4e07-96be-a14aa9c23543\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " Oct 31 12:58:23.754723 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.754232 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53634ad4-669e-4e07-96be-a14aa9c23543-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53634ad4-669e-4e07-96be-a14aa9c23543-plugins-config-volume\") pod \"53634ad4-669e-4e07-96be-a14aa9c23543\" (UID: \"53634ad4-669e-4e07-96be-a14aa9c23543\") " Oct 31 12:58:23.755867 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.755821 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53634ad4-669e-4e07-96be-a14aa9c23543-plugins-config-volume" pod "53634ad4-669e-4e07-96be-a14aa9c23543" (UID: "53634ad4-669e-4e07-96be-a14aa9c23543"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:23.765302 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.765233 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53634ad4-669e-4e07-96be-a14aa9c23543-kube-api-access-gc6f7" pod "53634ad4-669e-4e07-96be-a14aa9c23543" (UID: "53634ad4-669e-4e07-96be-a14aa9c23543"). InnerVolumeSpecName "kube-api-access-gc6f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:23.857348 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.857314 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc6f7\" (UniqueName: \"kubernetes.io/projected/53634ad4-669e-4e07-96be-a14aa9c23543-kube-api-access-gc6f7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:23.857348 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.857348 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53634ad4-669e-4e07-96be-a14aa9c23543-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:23.962420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.961832 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 12:58:23.962420 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:23.961884 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 12:58:24.466089 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.466030 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" event={"ID":"f78e6005-f207-488d-a0c4-3e783888b3b2","Type":"ContainerStarted","Data":"0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5"} Oct 31 12:58:24.468273 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.468249 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" Oct 31 12:58:24.472697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.468689 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5" event={"ID":"79246ba3-75f6-4afa-9ac0-9448e5bdf6f2","Type":"ContainerDied","Data":"72fae9edd7ab9d65eb9ed7e1736c80c9b99d92a33b7e03ec452d0c13bd23c343"} Oct 31 12:58:24.472697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.468753 5074 scope.go:122] "RemoveContainer" containerID="41139cf5a3be546c36e7d01332212bc3d16526a454a11bebfbd8da74394cc02c" Oct 31 12:58:24.497113 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.483498 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" event={"ID":"53634ad4-669e-4e07-96be-a14aa9c23543","Type":"ContainerDied","Data":"192c30da59227b546b31db0d2fb4faaee7ff22b372f00cf56c2bf63e95678d88"} Oct 31 12:58:24.497113 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.483620 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl" Oct 31 12:58:24.541364 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.536546 5074 scope.go:122] "RemoveContainer" containerID="ae00cad027a5cfee827e3c46f87de1df545251acc4dc684c817d2a3c4253e42e" Oct 31 12:58:24.547556 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.547516 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl"] Oct 31 12:58:24.552972 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.552946 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-702419-59d7c7fdf9-sw2sl"] Oct 31 12:58:24.565822 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.565798 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5"] Oct 31 12:58:24.610417 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.610376 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-702419-68cbcf79f7-2scn5"] Oct 31 12:58:24.961939 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.961871 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 12:58:24.962179 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:24.962157 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 12:58:25.392666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:25.390831 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:58:25.392666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:25.391953 5074 patch_prober.go:29] interesting pod/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 12:58:25.392666 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:25.392016 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 12:58:25.986969 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:25.986922 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53634ad4-669e-4e07-96be-a14aa9c23543" path="/var/lib/kubelet/pods/53634ad4-669e-4e07-96be-a14aa9c23543/volumes" Oct 31 12:58:25.993974 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:25.992134 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" path="/var/lib/kubelet/pods/79246ba3-75f6-4afa-9ac0-9448e5bdf6f2/volumes" Oct 31 12:58:26.040656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:25.997646 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 12:58:26.040966 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:26.040932 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 12:58:26.165581 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:58:26.165452 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:58:26.401657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:26.400343 5074 patch_prober.go:29] interesting pod/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 12:58:26.401657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:26.400405 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 12:58:26.593719 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:26.593115 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11391-a96ade" event={"ID":"2dc601c6-fd43-4a22-8abf-27356d92523d","Type":"ContainerStarted","Data":"343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f"} Oct 31 12:58:26.628678 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:26.628131 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11391-a96ade" podStartSLOduration=1.30678585 podStartE2EDuration="6.628111972s" totalImagesPullingTime="5.321326122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:26.625014981 +0000 UTC m=+1600.802732958" watchObservedRunningTime="2026-10-31 12:58:26.628111972 +0000 UTC m=+1600.805829956" Oct 31 12:58:26.962976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:26.962830 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 12:58:26.962976 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:26.962871 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 12:58:27.392657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:27.389996 5074 patch_prober.go:29] interesting pod/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 12:58:27.392657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:27.390079 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 12:58:27.427703 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:27.427670 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11391-a96ade"] Oct 31 12:58:27.436689 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:27.436655 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11391-a96ade"] Oct 31 12:58:28.217115 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.217063 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt"] Oct 31 12:58:28.217995 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.217963 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2dc601c6-fd43-4a22-8abf-27356d92523d" containerName="test-curl-11391-a96ade" Oct 31 12:58:28.217995 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.217990 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc601c6-fd43-4a22-8abf-27356d92523d" containerName="test-curl-11391-a96ade" Oct 31 12:58:28.218219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218041 5074 cpu_manager.go:414] "containerMap: removing container" podUID="53634ad4-669e-4e07-96be-a14aa9c23543" containerName="payload-pre-processing" Oct 31 12:58:28.218219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218049 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53634ad4-669e-4e07-96be-a14aa9c23543" containerName="payload-pre-processing" Oct 31 12:58:28.218219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218068 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f9ff4cad-7f28-4999-8fdd-c3c63ab58870" containerName="revoke-keys" Oct 31 12:58:28.218219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218076 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ff4cad-7f28-4999-8fdd-c3c63ab58870" containerName="revoke-keys" Oct 31 12:58:28.218219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218112 5074 cpu_manager.go:414] "containerMap: removing container" podUID="79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" containerName="payload-processing" Oct 31 12:58:28.218219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218119 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" containerName="payload-processing" Oct 31 12:58:28.218563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218270 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc601c6-fd43-4a22-8abf-27356d92523d" containerName="test-curl-11391-a96ade" Oct 31 12:58:28.218563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218288 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="53634ad4-669e-4e07-96be-a14aa9c23543" containerName="payload-pre-processing" Oct 31 12:58:28.218563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218299 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ff4cad-7f28-4999-8fdd-c3c63ab58870" containerName="revoke-keys" Oct 31 12:58:28.218563 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.218313 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="79246ba3-75f6-4afa-9ac0-9448e5bdf6f2" containerName="payload-processing" Oct 31 12:58:28.229036 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.229004 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.252028 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.251996 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:58:28.252297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.252258 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:58:28.252297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.252223 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:58:28.252297 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.252201 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:58:28.283612 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.283585 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt"] Oct 31 12:58:28.345377 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.345340 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.346456 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.346428 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.346714 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.346694 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbs5\" (UniqueName: \"kubernetes.io/projected/81fce511-14fd-46e8-84d5-ce15899a5f6c-kube-api-access-2vbs5\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.424429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.424392 5074 patch_prober.go:29] interesting pod/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 12:58:28.424747 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.424715 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 12:58:28.453428 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.453393 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.454104 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.453500 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbs5\" (UniqueName: \"kubernetes.io/projected/81fce511-14fd-46e8-84d5-ce15899a5f6c-kube-api-access-2vbs5\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.454104 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.453660 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.455380 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.455355 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.457342 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.457014 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.617733 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.615812 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbs5\" (UniqueName: \"kubernetes.io/projected/81fce511-14fd-46e8-84d5-ce15899a5f6c-kube-api-access-2vbs5\") pod \"maas-api-revoke-keys-e2e-mt-c4167730-7n8pt\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.655653 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.652873 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:28.843656 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.836721 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11391-a96ade" podUID="2dc601c6-fd43-4a22-8abf-27356d92523d" containerName="test-curl-11391-a96ade" containerID="cri-o://343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f" gracePeriod=2 Oct 31 12:58:28.936945 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.936862 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:28.938219 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.938069 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:28.939869 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:28.939814 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 12:58:29.253361 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:58:29.252437 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fce511_14fd_46e8_84d5_ce15899a5f6c.slice/crio-9522d5356abefeffdd11a1137a53d2ddf91e4177408f46bcad4be5ce41e29050 WatchSource:0}: Error finding container 9522d5356abefeffdd11a1137a53d2ddf91e4177408f46bcad4be5ce41e29050: Status 404 returned error can't find the container with id 9522d5356abefeffdd11a1137a53d2ddf91e4177408f46bcad4be5ce41e29050 Oct 31 12:58:29.260230 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:29.260206 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt"] Oct 31 12:58:29.390187 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:29.390148 5074 patch_prober.go:29] interesting pod/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 12:58:29.390422 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:29.390393 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 12:58:29.824255 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:29.824164 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" event={"ID":"81fce511-14fd-46e8-84d5-ce15899a5f6c","Type":"ContainerStarted","Data":"fda75ed246acc13487c0c97d56d287cd2b520b7b10ede71b0dca8bc2d80892fe"} Oct 31 12:58:29.824255 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:29.824226 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" event={"ID":"81fce511-14fd-46e8-84d5-ce15899a5f6c","Type":"ContainerStarted","Data":"9522d5356abefeffdd11a1137a53d2ddf91e4177408f46bcad4be5ce41e29050"} Oct 31 12:58:29.850819 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:29.850760 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" podStartSLOduration=1.850732398 podStartE2EDuration="1.850732398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:29.847612451 +0000 UTC m=+1604.025330439" watchObservedRunningTime="2026-10-31 12:58:29.850732398 +0000 UTC m=+1604.028450384" Oct 31 12:58:30.393908 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:30.393877 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:58:30.394388 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:30.394362 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:58:30.395096 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:30.395076 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:58:30.836914 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:30.836879 5074 generic.go:356] "Generic (PLEG): container finished" podID="81fce511-14fd-46e8-84d5-ce15899a5f6c" containerID="fda75ed246acc13487c0c97d56d287cd2b520b7b10ede71b0dca8bc2d80892fe" exitCode=0 Oct 31 12:58:30.837373 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:30.836964 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" event={"ID":"81fce511-14fd-46e8-84d5-ce15899a5f6c","Type":"ContainerDied","Data":"fda75ed246acc13487c0c97d56d287cd2b520b7b10ede71b0dca8bc2d80892fe"} Oct 31 12:58:31.157788 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.157762 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:31.286657 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.286111 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc601c6-fd43-4a22-8abf-27356d92523d-kube-api-access-4z2lh\" (UniqueName: \"kubernetes.io/projected/2dc601c6-fd43-4a22-8abf-27356d92523d-kube-api-access-4z2lh\") pod \"2dc601c6-fd43-4a22-8abf-27356d92523d\" (UID: \"2dc601c6-fd43-4a22-8abf-27356d92523d\") " Oct 31 12:58:31.299754 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.298904 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc601c6-fd43-4a22-8abf-27356d92523d-kube-api-access-4z2lh" pod "2dc601c6-fd43-4a22-8abf-27356d92523d" (UID: "2dc601c6-fd43-4a22-8abf-27356d92523d"). InnerVolumeSpecName "kube-api-access-4z2lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:31.389237 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.388821 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2lh\" (UniqueName: \"kubernetes.io/projected/2dc601c6-fd43-4a22-8abf-27356d92523d-kube-api-access-4z2lh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:31.848917 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.848882 5074 generic.go:356] "Generic (PLEG): container finished" podID="2dc601c6-fd43-4a22-8abf-27356d92523d" containerID="343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f" exitCode=137 Oct 31 12:58:31.850164 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.850137 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:31.850545 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.850525 5074 scope.go:122] "RemoveContainer" containerID="343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f" Oct 31 12:58:31.878462 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.878361 5074 scope.go:122] "RemoveContainer" containerID="343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f" Oct 31 12:58:31.878901 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:58:31.878871 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f\": container with ID starting with 343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f not found: ID does not exist" containerID="343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f" Oct 31 12:58:31.879054 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.878913 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f"} err="failed to get container status \"343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f\": rpc error: code = NotFound desc = could not find container \"343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f\": container with ID starting with 343b6f8ccfd9da38a876870b2f558065145968fb3949f8e607241f4a005bb42f not found: ID does not exist" Oct 31 12:58:31.882375 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:58:31.881518 5074 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11391-a96ade\" is forbidden: User \"system:node:ip-10-0-135-41.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-41.us-east-2.compute.internal' and this object" podUID="2dc601c6-fd43-4a22-8abf-27356d92523d" pod="opendatahub/test-curl-11391-a96ade" Oct 31 12:58:31.947676 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:31.946733 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc601c6-fd43-4a22-8abf-27356d92523d" path="/var/lib/kubelet/pods/2dc601c6-fd43-4a22-8abf-27356d92523d/volumes" Oct 31 12:58:32.082623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.082598 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:32.210055 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.209943 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fce511-14fd-46e8-84d5-ce15899a5f6c-kube-api-access-2vbs5\" (UniqueName: \"kubernetes.io/projected/81fce511-14fd-46e8-84d5-ce15899a5f6c-kube-api-access-2vbs5\") pod \"81fce511-14fd-46e8-84d5-ce15899a5f6c\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " Oct 31 12:58:32.210226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.210124 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-openshift-ca\") pod \"81fce511-14fd-46e8-84d5-ce15899a5f6c\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " Oct 31 12:58:32.210226 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.210184 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-xks-ca\") pod \"81fce511-14fd-46e8-84d5-ce15899a5f6c\" (UID: \"81fce511-14fd-46e8-84d5-ce15899a5f6c\") " Oct 31 12:58:32.210526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.210496 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-openshift-ca" pod "81fce511-14fd-46e8-84d5-ce15899a5f6c" (UID: "81fce511-14fd-46e8-84d5-ce15899a5f6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:32.210930 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.210905 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:32.212832 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.212804 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-xks-ca" pod "81fce511-14fd-46e8-84d5-ce15899a5f6c" (UID: "81fce511-14fd-46e8-84d5-ce15899a5f6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:32.212994 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.212971 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fce511-14fd-46e8-84d5-ce15899a5f6c-kube-api-access-2vbs5" pod "81fce511-14fd-46e8-84d5-ce15899a5f6c" (UID: "81fce511-14fd-46e8-84d5-ce15899a5f6c"). InnerVolumeSpecName "kube-api-access-2vbs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:32.312070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.312028 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vbs5\" (UniqueName: \"kubernetes.io/projected/81fce511-14fd-46e8-84d5-ce15899a5f6c-kube-api-access-2vbs5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:32.312070 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.312061 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fce511-14fd-46e8-84d5-ce15899a5f6c-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:32.859232 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.859204 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" Oct 31 12:58:32.859712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.859238 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt" event={"ID":"81fce511-14fd-46e8-84d5-ce15899a5f6c","Type":"ContainerDied","Data":"9522d5356abefeffdd11a1137a53d2ddf91e4177408f46bcad4be5ce41e29050"} Oct 31 12:58:32.859712 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:32.859299 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9522d5356abefeffdd11a1137a53d2ddf91e4177408f46bcad4be5ce41e29050" Oct 31 12:58:37.662904 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:37.662868 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:58:44.659449 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.659404 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw"] Oct 31 12:58:44.659950 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.659617 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" containerID="cri-o://0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5" gracePeriod=30 Oct 31 12:58:44.928740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.924571 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8"] Oct 31 12:58:44.928740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.925186 5074 cpu_manager.go:414] "containerMap: removing container" podUID="81fce511-14fd-46e8-84d5-ce15899a5f6c" containerName="revoke-keys" Oct 31 12:58:44.928740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.925202 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fce511-14fd-46e8-84d5-ce15899a5f6c" containerName="revoke-keys" Oct 31 12:58:44.928740 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.925323 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fce511-14fd-46e8-84d5-ce15899a5f6c" containerName="revoke-keys" Oct 31 12:58:44.934651 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.930131 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:44.935079 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.935055 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:58:44.939817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.935672 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:58:44.939817 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.938725 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8"] Oct 31 12:58:44.988811 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.988779 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:58:44.989039 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:44.988906 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:58:45.044034 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.044000 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.044246 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.044225 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.044322 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.044269 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpkl\" (UniqueName: \"kubernetes.io/projected/03c1356a-633f-42c4-8381-f8e5a6592f21-kube-api-access-7zpkl\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.145324 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.145290 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.145499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.145434 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.145499 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.145460 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpkl\" (UniqueName: \"kubernetes.io/projected/03c1356a-633f-42c4-8381-f8e5a6592f21-kube-api-access-7zpkl\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.145945 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.145921 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.147820 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.147799 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.158451 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.158429 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpkl\" (UniqueName: \"kubernetes.io/projected/03c1356a-633f-42c4-8381-f8e5a6592f21-kube-api-access-7zpkl\") pod \"maas-api-revoke-sub-a169d6be-j4xt8\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.260756 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.260704 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:45.398218 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.398159 5074 patch_prober.go:29] interesting pod/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 12:58:45.398218 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.398232 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 12:58:45.566418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:45.566383 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8"] Oct 31 12:58:45.567830 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:58:45.567798 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c1356a_633f_42c4_8381_f8e5a6592f21.slice/crio-1896d7ddfff9a01a7604946fa1ff9a8c3bfc1b6aa8cc307d65271d785fa8594e WatchSource:0}: Error finding container 1896d7ddfff9a01a7604946fa1ff9a8c3bfc1b6aa8cc307d65271d785fa8594e: Status 404 returned error can't find the container with id 1896d7ddfff9a01a7604946fa1ff9a8c3bfc1b6aa8cc307d65271d785fa8594e Oct 31 12:58:46.060199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.060158 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" event={"ID":"03c1356a-633f-42c4-8381-f8e5a6592f21","Type":"ContainerStarted","Data":"eedbfc89b34f1c1f9afe77eb7db28f7f304a012b63b7f445ffd3271bd7e255eb"} Oct 31 12:58:46.060581 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.060223 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" event={"ID":"03c1356a-633f-42c4-8381-f8e5a6592f21","Type":"ContainerStarted","Data":"1896d7ddfff9a01a7604946fa1ff9a8c3bfc1b6aa8cc307d65271d785fa8594e"} Oct 31 12:58:46.097342 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.097276 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" podStartSLOduration=2.097259106 podStartE2EDuration="2.097259106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:46.081729821 +0000 UTC m=+1620.259447805" watchObservedRunningTime="2026-10-31 12:58:46.097259106 +0000 UTC m=+1620.274977102" Oct 31 12:58:46.454218 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.454133 5074 scope.go:122] "RemoveContainer" containerID="9c72c143db34ee2ce34b55256df6b2f55fd0e6a55c153e724427303fba0d3d0a" Oct 31 12:58:46.464921 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.464894 5074 scope.go:122] "RemoveContainer" containerID="0ff5873f70665bf8de14c8c5ff0ab14537d14b22f360edb4396e2009115f1b90" Oct 31 12:58:46.474227 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.474210 5074 scope.go:122] "RemoveContainer" containerID="7c7f2a7341d494f5743c77e20aa90c15e990bb2dbdda2b52e3a59e94e9aedc51" Oct 31 12:58:46.486351 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.486330 5074 scope.go:122] "RemoveContainer" containerID="d4c30a4e83c831e13a3583bb5f5da2aff71e51783374f8c1336e5e52806b368f" Oct 31 12:58:46.497697 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:46.497674 5074 scope.go:122] "RemoveContainer" containerID="952e9d6a1df13205ccafbff2048708a873a905d9730ec0a988591b1361da03d6" Oct 31 12:58:47.069207 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:47.069176 5074 generic.go:356] "Generic (PLEG): container finished" podID="03c1356a-633f-42c4-8381-f8e5a6592f21" containerID="eedbfc89b34f1c1f9afe77eb7db28f7f304a012b63b7f445ffd3271bd7e255eb" exitCode=0 Oct 31 12:58:47.069558 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:47.069260 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" event={"ID":"03c1356a-633f-42c4-8381-f8e5a6592f21","Type":"ContainerDied","Data":"eedbfc89b34f1c1f9afe77eb7db28f7f304a012b63b7f445ffd3271bd7e255eb"} Oct 31 12:58:48.236467 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.236442 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:48.287206 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.287157 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-openshift-ca\") pod \"03c1356a-633f-42c4-8381-f8e5a6592f21\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " Oct 31 12:58:48.287620 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.287591 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-xks-ca\") pod \"03c1356a-633f-42c4-8381-f8e5a6592f21\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " Oct 31 12:58:48.287749 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.287707 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c1356a-633f-42c4-8381-f8e5a6592f21-kube-api-access-7zpkl\" (UniqueName: \"kubernetes.io/projected/03c1356a-633f-42c4-8381-f8e5a6592f21-kube-api-access-7zpkl\") pod \"03c1356a-633f-42c4-8381-f8e5a6592f21\" (UID: \"03c1356a-633f-42c4-8381-f8e5a6592f21\") " Oct 31 12:58:48.287968 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.287944 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-openshift-ca" pod "03c1356a-633f-42c4-8381-f8e5a6592f21" (UID: "03c1356a-633f-42c4-8381-f8e5a6592f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:48.288671 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.288606 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:48.291330 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.291295 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c1356a-633f-42c4-8381-f8e5a6592f21-kube-api-access-7zpkl" pod "03c1356a-633f-42c4-8381-f8e5a6592f21" (UID: "03c1356a-633f-42c4-8381-f8e5a6592f21"). InnerVolumeSpecName "kube-api-access-7zpkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:48.296475 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.296449 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-xks-ca" pod "03c1356a-633f-42c4-8381-f8e5a6592f21" (UID: "03c1356a-633f-42c4-8381-f8e5a6592f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:58:48.390594 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.390508 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c1356a-633f-42c4-8381-f8e5a6592f21-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:48.390594 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:48.390548 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zpkl\" (UniqueName: \"kubernetes.io/projected/03c1356a-633f-42c4-8381-f8e5a6592f21-kube-api-access-7zpkl\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:49.089782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:49.089749 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" Oct 31 12:58:49.090001 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:49.089792 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8" event={"ID":"03c1356a-633f-42c4-8381-f8e5a6592f21","Type":"ContainerDied","Data":"1896d7ddfff9a01a7604946fa1ff9a8c3bfc1b6aa8cc307d65271d785fa8594e"} Oct 31 12:58:49.090001 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:49.089841 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1896d7ddfff9a01a7604946fa1ff9a8c3bfc1b6aa8cc307d65271d785fa8594e" Oct 31 12:58:49.969891 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:49.969862 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:58:50.011263 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011221 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-socket\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011276 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-kube-api-access-knzwd\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-kube-api-access-knzwd\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011306 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-data\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011334 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-token\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-token\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011501 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-certs\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011608 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-podinfo\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011623 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011650 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-envoy\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011691 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78e6005-f207-488d-a0c4-3e783888b3b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78e6005-f207-488d-a0c4-3e783888b3b2-istiod-ca-cert\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.011851 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.011820 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-credential-socket\") pod \"f78e6005-f207-488d-a0c4-3e783888b3b2\" (UID: \"f78e6005-f207-488d-a0c4-3e783888b3b2\") " Oct 31 12:58:50.012182 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.012154 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-data" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:50.012816 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.012292 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-certs" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:50.012816 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.012618 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.012816 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.012661 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.013027 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.012848 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-credential-socket" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:50.013392 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.013122 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-socket" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:50.013716 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.013673 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78e6005-f207-488d-a0c4-3e783888b3b2-istiod-ca-cert" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:58:50.015510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.015486 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-podinfo" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 12:58:50.015613 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.015554 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-token" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:50.015985 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.015966 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-kube-api-access-knzwd" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "kube-api-access-knzwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:58:50.017224 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.017205 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-envoy" pod "f78e6005-f207-488d-a0c4-3e783888b3b2" (UID: "f78e6005-f207-488d-a0c4-3e783888b3b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:58:50.100189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.100151 5074 generic.go:356] "Generic (PLEG): container finished" podID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerID="0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5" exitCode=0 Oct 31 12:58:50.100343 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.100205 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" Oct 31 12:58:50.100343 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.100213 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" event={"ID":"f78e6005-f207-488d-a0c4-3e783888b3b2","Type":"ContainerDied","Data":"0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5"} Oct 31 12:58:50.100343 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.100268 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw" event={"ID":"f78e6005-f207-488d-a0c4-3e783888b3b2","Type":"ContainerDied","Data":"c2f9703a1b93e248be4eebddbd19cab25e2bd0c21b5d8ba668cc14cc2fdfaa0e"} Oct 31 12:58:50.100343 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.100296 5074 scope.go:122] "RemoveContainer" containerID="0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5" Oct 31 12:58:50.113696 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113666 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.113696 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113692 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.113803 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113704 5074 scope.go:122] "RemoveContainer" containerID="3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b" Oct 31 12:58:50.113803 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113706 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78e6005-f207-488d-a0c4-3e783888b3b2-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.113803 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113784 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.113920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113800 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78e6005-f207-488d-a0c4-3e783888b3b2-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.113920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113814 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knzwd\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-kube-api-access-knzwd\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.113920 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.113825 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78e6005-f207-488d-a0c4-3e783888b3b2-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:58:50.125526 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.125497 5074 scope.go:122] "RemoveContainer" containerID="0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5" Oct 31 12:58:50.125806 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:58:50.125779 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5\": container with ID starting with 0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5 not found: ID does not exist" containerID="0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5" Oct 31 12:58:50.125908 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.125817 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5"} err="failed to get container status \"0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5\": rpc error: code = NotFound desc = could not find container \"0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5\": container with ID starting with 0555a41bd2cea666e3eb18c491254a05c63fd865e8ae35d6055dc8c557e51de5 not found: ID does not exist" Oct 31 12:58:50.125908 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.125840 5074 scope.go:122] "RemoveContainer" containerID="3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b" Oct 31 12:58:50.126148 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:58:50.126126 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b\": container with ID starting with 3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b not found: ID does not exist" containerID="3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b" Oct 31 12:58:50.126210 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.126154 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b"} err="failed to get container status \"3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b\": rpc error: code = NotFound desc = could not find container \"3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b\": container with ID starting with 3e051b70242f33f265892882b665fae5e398bf8412010e801b0219106b2f466b not found: ID does not exist" Oct 31 12:58:50.126824 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.126804 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw"] Oct 31 12:58:50.147339 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:50.147316 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-73503b-openshift-default-64989dc4bc-7fjdw"] Oct 31 12:58:51.939311 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:51.939274 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" path="/var/lib/kubelet/pods/f78e6005-f207-488d-a0c4-3e783888b3b2/volumes" Oct 31 12:58:54.054646 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:54.054598 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw"] Oct 31 12:58:54.087510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:54.087473 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b56577a-dzhlw"] Oct 31 12:58:56.001382 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:56.001351 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9694848-827c-4b7a-ab7c-643aeef809d5" path="/var/lib/kubelet/pods/d9694848-827c-4b7a-ab7c-643aeef809d5/volumes" Oct 31 12:58:56.023408 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:56.023377 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt"] Oct 31 12:58:56.030888 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:56.030864 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c4167730-7n8pt"] Oct 31 12:58:56.038233 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:58:56.038187 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:58:57.944102 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:57.944064 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fce511-14fd-46e8-84d5-ce15899a5f6c" path="/var/lib/kubelet/pods/81fce511-14fd-46e8-84d5-ce15899a5f6c/volumes" Oct 31 12:58:59.703694 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:58:59.703662 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:59:01.224935 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.224905 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z"] Oct 31 12:59:01.225694 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225674 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" Oct 31 12:59:01.225808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225695 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" Oct 31 12:59:01.225808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225720 5074 cpu_manager.go:414] "containerMap: removing container" podUID="03c1356a-633f-42c4-8381-f8e5a6592f21" containerName="revoke-keys" Oct 31 12:59:01.225808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225727 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c1356a-633f-42c4-8381-f8e5a6592f21" containerName="revoke-keys" Oct 31 12:59:01.225808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225745 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" Oct 31 12:59:01.225808 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225754 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" Oct 31 12:59:01.226084 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225927 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" Oct 31 12:59:01.226084 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.225955 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c1356a-633f-42c4-8381-f8e5a6592f21" containerName="revoke-keys" Oct 31 12:59:01.231448 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.231427 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.245484 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.245458 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z"] Oct 31 12:59:01.249609 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.249578 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-21c556-dockercfg-nvrnl\"" Oct 31 12:59:01.252135 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.252099 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-21c556\"" Oct 31 12:59:01.341873 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.323648 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hk7\" (UniqueName: \"kubernetes.io/projected/73181174-8c6c-451b-b936-6b137ea0ea4b-kube-api-access-m8hk7\") pod \"payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.342286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.342252 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73181174-8c6c-451b-b936-6b137ea0ea4b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.452746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.450619 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hk7\" (UniqueName: \"kubernetes.io/projected/73181174-8c6c-451b-b936-6b137ea0ea4b-kube-api-access-m8hk7\") pod \"payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.452746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.450723 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73181174-8c6c-451b-b936-6b137ea0ea4b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.452746 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.452060 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73181174-8c6c-451b-b936-6b137ea0ea4b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.471909 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.471876 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hk7\" (UniqueName: \"kubernetes.io/projected/73181174-8c6c-451b-b936-6b137ea0ea4b-kube-api-access-m8hk7\") pod \"payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.548037 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.547960 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:01.552529 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.552503 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d"] Oct 31 12:59:01.553327 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.553304 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78e6005-f207-488d-a0c4-3e783888b3b2" containerName="istio-proxy" Oct 31 12:59:01.557531 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.557510 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.562949 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.562922 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:59:01.563616 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.563596 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:59:01.565767 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.565539 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:59:01.565767 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.565724 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:59:01.568799 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.568777 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d"] Oct 31 12:59:01.655486 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.654061 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.655486 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.654267 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6t7\" (UniqueName: \"kubernetes.io/projected/6b2f67af-fc90-40c8-873e-38b3f18fdc96-kube-api-access-dv6t7\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.655486 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.654326 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.756202 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.756168 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.756363 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.756333 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6t7\" (UniqueName: \"kubernetes.io/projected/6b2f67af-fc90-40c8-873e-38b3f18fdc96-kube-api-access-dv6t7\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.756405 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.756382 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.756936 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.756909 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.758806 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.758771 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.790382 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.790353 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6t7\" (UniqueName: \"kubernetes.io/projected/6b2f67af-fc90-40c8-873e-38b3f18fdc96-kube-api-access-dv6t7\") pod \"maas-api-revoke-sub-4677c662-79v4d\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:01.861190 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:59:01.861159 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73181174_8c6c_451b_b936_6b137ea0ea4b.slice/crio-63a52ad727388503c2d57af97ee7d31346ad25df9ae054d6edc655a9a1da2ebb WatchSource:0}: Error finding container 63a52ad727388503c2d57af97ee7d31346ad25df9ae054d6edc655a9a1da2ebb: Status 404 returned error can't find the container with id 63a52ad727388503c2d57af97ee7d31346ad25df9ae054d6edc655a9a1da2ebb Oct 31 12:59:01.862111 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.862087 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z"] Oct 31 12:59:01.919648 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:01.919609 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:02.205236 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:59:02.205205 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2f67af_fc90_40c8_873e_38b3f18fdc96.slice/crio-c744f8038b4cf87e36655b83a248a1df1c06b9dfb743f00f0916bdb9f1a86bf4 WatchSource:0}: Error finding container c744f8038b4cf87e36655b83a248a1df1c06b9dfb743f00f0916bdb9f1a86bf4: Status 404 returned error can't find the container with id c744f8038b4cf87e36655b83a248a1df1c06b9dfb743f00f0916bdb9f1a86bf4 Oct 31 12:59:02.205776 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:02.205750 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d"] Oct 31 12:59:02.295886 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:02.295852 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" event={"ID":"6b2f67af-fc90-40c8-873e-38b3f18fdc96","Type":"ContainerStarted","Data":"c744f8038b4cf87e36655b83a248a1df1c06b9dfb743f00f0916bdb9f1a86bf4"} Oct 31 12:59:02.297658 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:02.297603 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" event={"ID":"73181174-8c6c-451b-b936-6b137ea0ea4b","Type":"ContainerStarted","Data":"de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62"} Oct 31 12:59:02.297768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:02.297680 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" event={"ID":"73181174-8c6c-451b-b936-6b137ea0ea4b","Type":"ContainerStarted","Data":"63a52ad727388503c2d57af97ee7d31346ad25df9ae054d6edc655a9a1da2ebb"} Oct 31 12:59:02.297768 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:02.297710 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:02.321983 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:02.321921 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" podStartSLOduration=1.321908578 podStartE2EDuration="1.321908578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:02.319330129 +0000 UTC m=+1636.497048110" watchObservedRunningTime="2026-10-31 12:59:02.321908578 +0000 UTC m=+1636.499626599" Oct 31 12:59:03.307189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:03.307156 5074 generic.go:356] "Generic (PLEG): container finished" podID="6b2f67af-fc90-40c8-873e-38b3f18fdc96" containerID="3d0802d4280112f499df31c3d0d317455f2a463a782fc6a2945ed601fb7ca29f" exitCode=0 Oct 31 12:59:03.307574 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:03.307248 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" event={"ID":"6b2f67af-fc90-40c8-873e-38b3f18fdc96","Type":"ContainerDied","Data":"3d0802d4280112f499df31c3d0d317455f2a463a782fc6a2945ed601fb7ca29f"} Oct 31 12:59:04.661782 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.661754 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:04.825454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.825423 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2f67af-fc90-40c8-873e-38b3f18fdc96-kube-api-access-dv6t7\" (UniqueName: \"kubernetes.io/projected/6b2f67af-fc90-40c8-873e-38b3f18fdc96-kube-api-access-dv6t7\") pod \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " Oct 31 12:59:04.825617 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.825578 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-openshift-ca\") pod \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " Oct 31 12:59:04.825686 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.825673 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-xks-ca\") pod \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\" (UID: \"6b2f67af-fc90-40c8-873e-38b3f18fdc96\") " Oct 31 12:59:04.826001 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.825955 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-openshift-ca" pod "6b2f67af-fc90-40c8-873e-38b3f18fdc96" (UID: "6b2f67af-fc90-40c8-873e-38b3f18fdc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:04.826228 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.826207 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:04.827598 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.827572 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2f67af-fc90-40c8-873e-38b3f18fdc96-kube-api-access-dv6t7" pod "6b2f67af-fc90-40c8-873e-38b3f18fdc96" (UID: "6b2f67af-fc90-40c8-873e-38b3f18fdc96"). InnerVolumeSpecName "kube-api-access-dv6t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:04.827598 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.827580 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-xks-ca" pod "6b2f67af-fc90-40c8-873e-38b3f18fdc96" (UID: "6b2f67af-fc90-40c8-873e-38b3f18fdc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:04.927627 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.927549 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f67af-fc90-40c8-873e-38b3f18fdc96-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:04.927627 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:04.927579 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6t7\" (UniqueName: \"kubernetes.io/projected/6b2f67af-fc90-40c8-873e-38b3f18fdc96-kube-api-access-dv6t7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:05.325416 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:05.325386 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" Oct 31 12:59:05.325593 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:05.325415 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d" event={"ID":"6b2f67af-fc90-40c8-873e-38b3f18fdc96","Type":"ContainerDied","Data":"c744f8038b4cf87e36655b83a248a1df1c06b9dfb743f00f0916bdb9f1a86bf4"} Oct 31 12:59:05.325593 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:05.325461 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c744f8038b4cf87e36655b83a248a1df1c06b9dfb743f00f0916bdb9f1a86bf4" Oct 31 12:59:07.026469 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:07.026432 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65"] Oct 31 12:59:07.031094 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:07.031065 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef4914d-r9f65"] Oct 31 12:59:07.940286 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:07.940251 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93343735-2894-462b-a217-becdeddb3956" path="/var/lib/kubelet/pods/93343735-2894-462b-a217-becdeddb3956/volumes" Oct 31 12:59:08.025745 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:08.025708 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn"] Oct 31 12:59:08.030601 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:08.030570 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77626db3-l4fsn"] Oct 31 12:59:08.034481 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:08.034457 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955"] Oct 31 12:59:08.038095 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:08.038073 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2846bc8-qm955"] Oct 31 12:59:09.964964 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:09.964924 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5dd1ff-e593-4d4b-9730-f879c6f6f226" path="/var/lib/kubelet/pods/5a5dd1ff-e593-4d4b-9730-f879c6f6f226/volumes" Oct 31 12:59:09.965611 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:09.965589 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b32da8-638e-4150-bd3a-bb93ec713f9b" path="/var/lib/kubelet/pods/75b32da8-638e-4150-bd3a-bb93ec713f9b/volumes" Oct 31 12:59:10.727893 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.727859 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5"] Oct 31 12:59:10.728579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.728553 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6b2f67af-fc90-40c8-873e-38b3f18fdc96" containerName="revoke-keys" Oct 31 12:59:10.728579 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.728577 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2f67af-fc90-40c8-873e-38b3f18fdc96" containerName="revoke-keys" Oct 31 12:59:10.728794 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.728702 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2f67af-fc90-40c8-873e-38b3f18fdc96" containerName="revoke-keys" Oct 31 12:59:10.733846 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.733823 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.740879 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.740855 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 12:59:10.741029 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.740912 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 12:59:10.741029 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.740953 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 12:59:10.741494 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.741466 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xtsw2\"" Oct 31 12:59:10.741682 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.741513 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5"] Oct 31 12:59:10.813617 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.813574 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.813874 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.813656 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/070fa1fe-3a61-4c63-8566-c87f4d999aff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.813874 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.813718 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.813874 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.813767 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.814092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.813907 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.814092 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.813942 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.814209 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.814096 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52n2\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-kube-api-access-j52n2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.915150 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.915117 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.915336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.915194 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52n2\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-kube-api-access-j52n2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.915336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.915235 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.915336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.915294 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/070fa1fe-3a61-4c63-8566-c87f4d999aff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.915336 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.915331 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.915588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.915370 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.915588 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.915534 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.916429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.916387 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.917959 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.917926 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.918245 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.918220 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.918533 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.918486 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/070fa1fe-3a61-4c63-8566-c87f4d999aff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.921971 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.921945 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.928083 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.928054 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52n2\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-kube-api-access-j52n2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:10.929078 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:10.929057 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7lb5\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:11.072708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:11.071508 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:11.388890 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:59:11.388858 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070fa1fe_3a61_4c63_8566_c87f4d999aff.slice/crio-185e902d621bd902205440fd4a9e9c02671ef67c0fa1c7aed86494bcc213b170 WatchSource:0}: Error finding container 185e902d621bd902205440fd4a9e9c02671ef67c0fa1c7aed86494bcc213b170: Status 404 returned error can't find the container with id 185e902d621bd902205440fd4a9e9c02671ef67c0fa1c7aed86494bcc213b170 Oct 31 12:59:11.389520 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:11.389491 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5"] Oct 31 12:59:12.388418 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:12.388383 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" event={"ID":"070fa1fe-3a61-4c63-8566-c87f4d999aff","Type":"ContainerStarted","Data":"185e902d621bd902205440fd4a9e9c02671ef67c0fa1c7aed86494bcc213b170"} Oct 31 12:59:13.311402 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:13.311373 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:15.114729 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:15.114689 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:59:15.115086 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:15.114763 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 12:59:15.418283 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:15.418186 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" event={"ID":"070fa1fe-3a61-4c63-8566-c87f4d999aff","Type":"ContainerStarted","Data":"feac327767c27bc8d95df2da0a205fc954f51821e0ad64700ad420db5d0d0229"} Oct 31 12:59:15.418439 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:15.418286 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:15.449546 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:15.449491 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" podStartSLOduration=1.7268558459999999 podStartE2EDuration="5.44947684s" totalImagesPullingTime="3.722620994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:15.446008462 +0000 UTC m=+1649.623726443" watchObservedRunningTime="2026-10-31 12:59:15.44947684 +0000 UTC m=+1649.627194822" Oct 31 12:59:16.433153 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:16.433108 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 12:59:21.027975 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:21.027936 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc"] Oct 31 12:59:21.032371 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:21.032335 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9bc428-xtsnc"] Oct 31 12:59:21.939383 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:21.939349 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1997fa-4ab2-4892-ad00-48e874340162" path="/var/lib/kubelet/pods/9c1997fa-4ab2-4892-ad00-48e874340162/volumes" Oct 31 12:59:22.025369 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:22.025326 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt"] Oct 31 12:59:22.033573 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:22.033534 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16041536-6dzmt"] Oct 31 12:59:23.963614 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:23.963580 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44323960-f9ed-489c-8327-7ed50a173eab" path="/var/lib/kubelet/pods/44323960-f9ed-489c-8327-7ed50a173eab/volumes" Oct 31 12:59:24.135725 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:24.135682 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp"] Oct 31 12:59:24.149400 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:24.146806 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b20b56c-2qnzp"] Oct 31 12:59:25.940813 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:25.940776 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed772b52-b9a0-438b-b062-d6e0eaeb5aa8" path="/var/lib/kubelet/pods/ed772b52-b9a0-438b-b062-d6e0eaeb5aa8/volumes" Oct 31 12:59:25.984158 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:59:25.984127 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:59:35.027925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:35.027887 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn"] Oct 31 12:59:35.034451 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:35.034423 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b044ea-52qhn"] Oct 31 12:59:35.374470 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:35.374377 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff"] Oct 31 12:59:35.946515 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:35.946465 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc782c36-7587-4c5c-b8cb-435c18b86e2c" path="/var/lib/kubelet/pods/cc782c36-7587-4c5c-b8cb-435c18b86e2c/volumes" Oct 31 12:59:36.633189 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:36.633152 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 12:59:42.215429 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:42.215383 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z"] Oct 31 12:59:47.216533 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.216494 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" podUID="73181174-8c6c-451b-b936-6b137ea0ea4b" containerName="payload-processing" containerID="cri-o://de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62" gracePeriod=25 Oct 31 12:59:47.514318 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.514295 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:47.682483 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.682453 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73181174-8c6c-451b-b936-6b137ea0ea4b-kube-api-access-m8hk7\" (UniqueName: \"kubernetes.io/projected/73181174-8c6c-451b-b936-6b137ea0ea4b-kube-api-access-m8hk7\") pod \"73181174-8c6c-451b-b936-6b137ea0ea4b\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " Oct 31 12:59:47.682807 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.682583 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73181174-8c6c-451b-b936-6b137ea0ea4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73181174-8c6c-451b-b936-6b137ea0ea4b-plugins-config-volume\") pod \"73181174-8c6c-451b-b936-6b137ea0ea4b\" (UID: \"73181174-8c6c-451b-b936-6b137ea0ea4b\") " Oct 31 12:59:47.683019 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.682993 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73181174-8c6c-451b-b936-6b137ea0ea4b-plugins-config-volume" pod "73181174-8c6c-451b-b936-6b137ea0ea4b" (UID: "73181174-8c6c-451b-b936-6b137ea0ea4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:47.684715 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.684684 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73181174-8c6c-451b-b936-6b137ea0ea4b-kube-api-access-m8hk7" pod "73181174-8c6c-451b-b936-6b137ea0ea4b" (UID: "73181174-8c6c-451b-b936-6b137ea0ea4b"). InnerVolumeSpecName "kube-api-access-m8hk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:47.784520 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.784436 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73181174-8c6c-451b-b936-6b137ea0ea4b-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:47.784520 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.784474 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hk7\" (UniqueName: \"kubernetes.io/projected/73181174-8c6c-451b-b936-6b137ea0ea4b-kube-api-access-m8hk7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:47.795772 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.795742 5074 generic.go:356] "Generic (PLEG): container finished" podID="73181174-8c6c-451b-b936-6b137ea0ea4b" containerID="de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62" exitCode=0 Oct 31 12:59:47.795925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.795800 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" Oct 31 12:59:47.795925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.795830 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" event={"ID":"73181174-8c6c-451b-b936-6b137ea0ea4b","Type":"ContainerDied","Data":"de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62"} Oct 31 12:59:47.795925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.795889 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z" event={"ID":"73181174-8c6c-451b-b936-6b137ea0ea4b","Type":"ContainerDied","Data":"63a52ad727388503c2d57af97ee7d31346ad25df9ae054d6edc655a9a1da2ebb"} Oct 31 12:59:47.795925 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.795912 5074 scope.go:122] "RemoveContainer" containerID="de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62" Oct 31 12:59:47.812940 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.812893 5074 scope.go:122] "RemoveContainer" containerID="de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62" Oct 31 12:59:47.813270 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:59:47.813243 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62\": container with ID starting with de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62 not found: ID does not exist" containerID="de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62" Oct 31 12:59:47.813357 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.813283 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62"} err="failed to get container status \"de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62\": rpc error: code = NotFound desc = could not find container \"de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62\": container with ID starting with de64e1c0aa4ad54463c1db0bab2e6983727ca7dda2b54f5ace2909fe87dbad62 not found: ID does not exist" Oct 31 12:59:47.832055 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.832033 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z"] Oct 31 12:59:47.840603 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.840580 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21c556-7cbc4c79b9-6z77z"] Oct 31 12:59:47.939754 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:47.939724 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73181174-8c6c-451b-b936-6b137ea0ea4b" path="/var/lib/kubelet/pods/73181174-8c6c-451b-b936-6b137ea0ea4b/volumes" Oct 31 12:59:50.401830 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.401713 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" podUID="15589bf4-60b5-4b6f-ae3f-114cba7703ef" containerName="main" containerID="cri-o://0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318" gracePeriod=45 Oct 31 12:59:50.701700 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.699628 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:59:50.733296 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733263 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-home\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-home\") pod \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " Oct 31 12:59:50.733424 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733330 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15589bf4-60b5-4b6f-ae3f-114cba7703ef-kube-api-access-898bz\" (UniqueName: \"kubernetes.io/projected/15589bf4-60b5-4b6f-ae3f-114cba7703ef-kube-api-access-898bz\") pod \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " Oct 31 12:59:50.733424 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733390 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tmp-dir\") pod \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " Oct 31 12:59:50.733544 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733504 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-home" pod "15589bf4-60b5-4b6f-ae3f-114cba7703ef" (UID: "15589bf4-60b5-4b6f-ae3f-114cba7703ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:50.733544 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733516 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-dshm\") pod \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " Oct 31 12:59:50.733718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733697 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tmp-dir" pod "15589bf4-60b5-4b6f-ae3f-114cba7703ef" (UID: "15589bf4-60b5-4b6f-ae3f-114cba7703ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:50.733718 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733708 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-model-cache\") pod \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " Oct 31 12:59:50.733811 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733741 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tls-certs\") pod \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\" (UID: \"15589bf4-60b5-4b6f-ae3f-114cba7703ef\") " Oct 31 12:59:50.733982 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.733924 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-model-cache" pod "15589bf4-60b5-4b6f-ae3f-114cba7703ef" (UID: "15589bf4-60b5-4b6f-ae3f-114cba7703ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:50.734454 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.734429 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:50.734553 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.734457 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:50.734553 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.734471 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:50.735934 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.735910 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tls-certs" pod "15589bf4-60b5-4b6f-ae3f-114cba7703ef" (UID: "15589bf4-60b5-4b6f-ae3f-114cba7703ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:50.736041 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.736011 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-dshm" pod "15589bf4-60b5-4b6f-ae3f-114cba7703ef" (UID: "15589bf4-60b5-4b6f-ae3f-114cba7703ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 12:59:50.736100 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.736039 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15589bf4-60b5-4b6f-ae3f-114cba7703ef-kube-api-access-898bz" pod "15589bf4-60b5-4b6f-ae3f-114cba7703ef" (UID: "15589bf4-60b5-4b6f-ae3f-114cba7703ef"). InnerVolumeSpecName "kube-api-access-898bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:50.837104 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.835794 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15589bf4-60b5-4b6f-ae3f-114cba7703ef-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:50.837104 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.835824 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15589bf4-60b5-4b6f-ae3f-114cba7703ef-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:50.837104 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.835835 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-898bz\" (UniqueName: \"kubernetes.io/projected/15589bf4-60b5-4b6f-ae3f-114cba7703ef-kube-api-access-898bz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:50.852179 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.852143 5074 generic.go:356] "Generic (PLEG): container finished" podID="15589bf4-60b5-4b6f-ae3f-114cba7703ef" containerID="0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318" exitCode=0 Oct 31 12:59:50.852366 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.852348 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" event={"ID":"15589bf4-60b5-4b6f-ae3f-114cba7703ef","Type":"ContainerDied","Data":"0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318"} Oct 31 12:59:50.852438 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.852388 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" event={"ID":"15589bf4-60b5-4b6f-ae3f-114cba7703ef","Type":"ContainerDied","Data":"8011b9230daa7a6d9e3833dad82ea365cf3bdefc4c21e657cc280d3059e69142"} Oct 31 12:59:50.852438 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.852408 5074 scope.go:122] "RemoveContainer" containerID="0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318" Oct 31 12:59:50.852538 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.852515 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff" Oct 31 12:59:50.870789 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.870720 5074 scope.go:122] "RemoveContainer" containerID="0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318" Oct 31 12:59:50.871083 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:59:50.871022 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318\": container with ID starting with 0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318 not found: ID does not exist" containerID="0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318" Oct 31 12:59:50.871083 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.871060 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318"} err="failed to get container status \"0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318\": rpc error: code = NotFound desc = could not find container \"0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318\": container with ID starting with 0bb37b1e67b0d1eda2c530acbc3b8da6b5320e0c4d01fab0e01b2e646b81b318 not found: ID does not exist" Oct 31 12:59:50.885989 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.885954 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff"] Oct 31 12:59:50.899438 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:50.899412 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c4167730/e2e-lifecycle-model-c4167730-kserve-5b98dffcd9-pwkff"] Oct 31 12:59:51.939708 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:51.939677 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15589bf4-60b5-4b6f-ae3f-114cba7703ef" path="/var/lib/kubelet/pods/15589bf4-60b5-4b6f-ae3f-114cba7703ef/volumes" Oct 31 12:59:52.664958 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.664927 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t"] Oct 31 12:59:52.665510 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.665496 5074 cpu_manager.go:414] "containerMap: removing container" podUID="15589bf4-60b5-4b6f-ae3f-114cba7703ef" containerName="main" Oct 31 12:59:52.665561 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.665512 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15589bf4-60b5-4b6f-ae3f-114cba7703ef" containerName="main" Oct 31 12:59:52.665561 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.665527 5074 cpu_manager.go:414] "containerMap: removing container" podUID="73181174-8c6c-451b-b936-6b137ea0ea4b" containerName="payload-processing" Oct 31 12:59:52.665561 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.665532 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73181174-8c6c-451b-b936-6b137ea0ea4b" containerName="payload-processing" Oct 31 12:59:52.665693 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.665654 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="73181174-8c6c-451b-b936-6b137ea0ea4b" containerName="payload-processing" Oct 31 12:59:52.665693 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.665673 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="15589bf4-60b5-4b6f-ae3f-114cba7703ef" containerName="main" Oct 31 12:59:52.671199 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.671179 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.675907 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.675881 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 12:59:52.676396 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.676375 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 12:59:52.679702 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.679677 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t"] Oct 31 12:59:52.708900 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.708878 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 12:59:52.709033 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.708996 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 12:59:52.761615 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.761583 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6ph\" (UniqueName: \"kubernetes.io/projected/1771e040-f7a3-44ee-8ae0-f460478a74bf-kube-api-access-hx6ph\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.761798 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.761658 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.761880 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.761858 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.862733 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.862699 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.862928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.862822 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6ph\" (UniqueName: \"kubernetes.io/projected/1771e040-f7a3-44ee-8ae0-f460478a74bf-kube-api-access-hx6ph\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.862928 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.862862 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.863460 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.863438 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.865285 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.865263 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:52.875534 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:52.875509 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6ph\" (UniqueName: \"kubernetes.io/projected/1771e040-f7a3-44ee-8ae0-f460478a74bf-kube-api-access-hx6ph\") pod \"maas-api-revoke-sub-d92a1c0b-96t6t\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:53.002728 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:53.002700 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:53.292662 ip-10-0-135-41 kubenswrapper[5074]: W1031 12:59:53.292614 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1771e040_f7a3_44ee_8ae0_f460478a74bf.slice/crio-05a01ff18c2abbdb115b8aaa08d39db81f909067a4ad5071b9117d30f65d797e WatchSource:0}: Error finding container 05a01ff18c2abbdb115b8aaa08d39db81f909067a4ad5071b9117d30f65d797e: Status 404 returned error can't find the container with id 05a01ff18c2abbdb115b8aaa08d39db81f909067a4ad5071b9117d30f65d797e Oct 31 12:59:53.294811 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:53.294561 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t"] Oct 31 12:59:53.907251 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:53.907157 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" event={"ID":"1771e040-f7a3-44ee-8ae0-f460478a74bf","Type":"ContainerStarted","Data":"72787591a08e59ab4ea43c0aa69288013358771d0e5c600b91f795eb0c1e8e8a"} Oct 31 12:59:53.907251 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:53.907224 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" event={"ID":"1771e040-f7a3-44ee-8ae0-f460478a74bf","Type":"ContainerStarted","Data":"05a01ff18c2abbdb115b8aaa08d39db81f909067a4ad5071b9117d30f65d797e"} Oct 31 12:59:53.932758 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:53.932701 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" podStartSLOduration=1.9326862409999999 podStartE2EDuration="1.932686241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:53.926096225 +0000 UTC m=+1688.103814211" watchObservedRunningTime="2026-10-31 12:59:53.932686241 +0000 UTC m=+1688.110404224" Oct 31 12:59:54.916443 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:54.916403 5074 generic.go:356] "Generic (PLEG): container finished" podID="1771e040-f7a3-44ee-8ae0-f460478a74bf" containerID="72787591a08e59ab4ea43c0aa69288013358771d0e5c600b91f795eb0c1e8e8a" exitCode=0 Oct 31 12:59:54.916862 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:54.916476 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" event={"ID":"1771e040-f7a3-44ee-8ae0-f460478a74bf","Type":"ContainerDied","Data":"72787591a08e59ab4ea43c0aa69288013358771d0e5c600b91f795eb0c1e8e8a"} Oct 31 12:59:55.986523 ip-10-0-135-41 kubenswrapper[5074]: E1031 12:59:55.986479 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 12:59:56.103106 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.103080 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:56.220376 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.220347 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1771e040-f7a3-44ee-8ae0-f460478a74bf-kube-api-access-hx6ph\" (UniqueName: \"kubernetes.io/projected/1771e040-f7a3-44ee-8ae0-f460478a74bf-kube-api-access-hx6ph\") pod \"1771e040-f7a3-44ee-8ae0-f460478a74bf\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " Oct 31 12:59:56.220567 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.220527 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-xks-ca\") pod \"1771e040-f7a3-44ee-8ae0-f460478a74bf\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " Oct 31 12:59:56.220979 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.220953 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-openshift-ca\") pod \"1771e040-f7a3-44ee-8ae0-f460478a74bf\" (UID: \"1771e040-f7a3-44ee-8ae0-f460478a74bf\") " Oct 31 12:59:56.222285 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.222255 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-openshift-ca" pod "1771e040-f7a3-44ee-8ae0-f460478a74bf" (UID: "1771e040-f7a3-44ee-8ae0-f460478a74bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 12:59:56.223284 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.223263 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.231246 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.231189 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-xks-ca" pod "1771e040-f7a3-44ee-8ae0-f460478a74bf" (UID: "1771e040-f7a3-44ee-8ae0-f460478a74bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 12:59:56.232460 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.232433 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1771e040-f7a3-44ee-8ae0-f460478a74bf-kube-api-access-hx6ph" pod "1771e040-f7a3-44ee-8ae0-f460478a74bf" (UID: "1771e040-f7a3-44ee-8ae0-f460478a74bf"). InnerVolumeSpecName "kube-api-access-hx6ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 12:59:56.325196 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.325167 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1771e040-f7a3-44ee-8ae0-f460478a74bf-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.325337 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.325200 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6ph\" (UniqueName: \"kubernetes.io/projected/1771e040-f7a3-44ee-8ae0-f460478a74bf-kube-api-access-hx6ph\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 12:59:56.932371 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.932336 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" Oct 31 12:59:56.932371 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.932363 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t" event={"ID":"1771e040-f7a3-44ee-8ae0-f460478a74bf","Type":"ContainerDied","Data":"05a01ff18c2abbdb115b8aaa08d39db81f909067a4ad5071b9117d30f65d797e"} Oct 31 12:59:56.932613 ip-10-0-135-41 kubenswrapper[5074]: I1031 12:59:56.932415 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a01ff18c2abbdb115b8aaa08d39db81f909067a4ad5071b9117d30f65d797e" Oct 31 13:00:00.161773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.161736 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5"] Oct 31 13:00:00.162781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.162755 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1771e040-f7a3-44ee-8ae0-f460478a74bf" containerName="revoke-keys" Oct 31 13:00:00.162781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.162782 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1771e040-f7a3-44ee-8ae0-f460478a74bf" containerName="revoke-keys" Oct 31 13:00:00.162950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.162942 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1771e040-f7a3-44ee-8ae0-f460478a74bf" containerName="revoke-keys" Oct 31 13:00:00.167431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.167410 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.169992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.169954 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c"] Oct 31 13:00:00.179165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.179142 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7"] Oct 31 13:00:00.179480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.179458 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.183354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.183327 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:00:00.183468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.183431 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:00:00.184978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.184955 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.188106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.188085 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55"] Oct 31 13:00:00.193830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.192923 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.198479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.198276 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c"] Oct 31 13:00:00.200695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.200673 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:00:00.205557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.205474 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5"] Oct 31 13:00:00.223879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.222726 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7"] Oct 31 13:00:00.240737 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.240705 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55"] Oct 31 13:00:00.295665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.295592 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdpn\" (UniqueName: \"kubernetes.io/projected/41153fe2-e960-4866-a7f3-737ec64470d8-kube-api-access-frdpn\") pod \"maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.295853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.295714 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.295853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.295783 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.295990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.295973 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41153fe2-e960-4866-a7f3-737ec64470d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.296115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.296099 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfbm6\" (UniqueName: \"kubernetes.io/projected/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-kube-api-access-xfbm6\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.296178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.296145 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47acb20a-2db0-487a-8a57-2871d3ac58d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.296226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.296180 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbgn\" (UniqueName: \"kubernetes.io/projected/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-kube-api-access-mdbgn\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.296226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.296213 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjjl\" (UniqueName: \"kubernetes.io/projected/47acb20a-2db0-487a-8a57-2871d3ac58d0-kube-api-access-gjjjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.397736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.397689 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47acb20a-2db0-487a-8a57-2871d3ac58d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.397941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.397769 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbgn\" (UniqueName: \"kubernetes.io/projected/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-kube-api-access-mdbgn\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.397941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.397799 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjjl\" (UniqueName: \"kubernetes.io/projected/47acb20a-2db0-487a-8a57-2871d3ac58d0-kube-api-access-gjjjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.397941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.397847 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdpn\" (UniqueName: \"kubernetes.io/projected/41153fe2-e960-4866-a7f3-737ec64470d8-kube-api-access-frdpn\") pod \"maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.397941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.397893 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.398179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.397946 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.398179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.398148 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41153fe2-e960-4866-a7f3-737ec64470d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.398289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.398268 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfbm6\" (UniqueName: \"kubernetes.io/projected/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-kube-api-access-xfbm6\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.399196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.398534 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47acb20a-2db0-487a-8a57-2871d3ac58d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.399196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.398957 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.399196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.399149 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41153fe2-e960-4866-a7f3-737ec64470d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.399476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.399289 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.414907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.414836 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdpn\" (UniqueName: \"kubernetes.io/projected/41153fe2-e960-4866-a7f3-737ec64470d8-kube-api-access-frdpn\") pod \"maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.415055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.414947 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbgn\" (UniqueName: \"kubernetes.io/projected/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-kube-api-access-mdbgn\") pod \"maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.415055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.414956 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfbm6\" (UniqueName: \"kubernetes.io/projected/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-kube-api-access-xfbm6\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.416390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.416362 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjjl\" (UniqueName: \"kubernetes.io/projected/47acb20a-2db0-487a-8a57-2871d3ac58d0-kube-api-access-gjjjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.487303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.487263 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:00.505350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.505319 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:00.540872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.540844 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:00.547851 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.545834 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:00.831700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.831508 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5"] Oct 31 13:00:00.833211 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:00.832680 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb71e1a0_3a9e_4dd0_be4f_736c08d62d5f.slice/crio-c8b08082af3d5b5c1e67bf268923572cf110a3c438d3d177f21a6b73292dac38 WatchSource:0}: Error finding container c8b08082af3d5b5c1e67bf268923572cf110a3c438d3d177f21a6b73292dac38: Status 404 returned error can't find the container with id c8b08082af3d5b5c1e67bf268923572cf110a3c438d3d177f21a6b73292dac38 Oct 31 13:00:00.973532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.973502 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c"] Oct 31 13:00:00.987178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:00.987151 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerStarted","Data":"c8b08082af3d5b5c1e67bf268923572cf110a3c438d3d177f21a6b73292dac38"} Oct 31 13:00:01.006859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:01.006834 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7"] Oct 31 13:00:01.013907 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:01.013880 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41153fe2_e960_4866_a7f3_737ec64470d8.slice/crio-8b36090ec84876f865c167fd1111cdc003baef2850f6d564292d8e728d08f30e WatchSource:0}: Error finding container 8b36090ec84876f865c167fd1111cdc003baef2850f6d564292d8e728d08f30e: Status 404 returned error can't find the container with id 8b36090ec84876f865c167fd1111cdc003baef2850f6d564292d8e728d08f30e Oct 31 13:00:01.036528 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:01.036498 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b65a0a_934e_4fc6_8a3a_8d467d3fd8f1.slice/crio-1dadf3ac1fae4ee3f2ce9ee9d0ccb40afbc1beda83666714301eb18007dfe05d WatchSource:0}: Error finding container 1dadf3ac1fae4ee3f2ce9ee9d0ccb40afbc1beda83666714301eb18007dfe05d: Status 404 returned error can't find the container with id 1dadf3ac1fae4ee3f2ce9ee9d0ccb40afbc1beda83666714301eb18007dfe05d Oct 31 13:00:01.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:01.037505 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55"] Oct 31 13:00:02.000435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.000396 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerStarted","Data":"636ddb2a30a85c5c4248b21f5030d4ecec937dd7dc13c79db4224a07215fb11f"} Oct 31 13:00:02.002571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.002542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" event={"ID":"41153fe2-e960-4866-a7f3-737ec64470d8","Type":"ContainerStarted","Data":"2a9106fc1b3a835140569783f0b48737d5873a011eae5812943db02e9d72a446"} Oct 31 13:00:02.002730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.002592 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" event={"ID":"41153fe2-e960-4866-a7f3-737ec64470d8","Type":"ContainerStarted","Data":"8b36090ec84876f865c167fd1111cdc003baef2850f6d564292d8e728d08f30e"} Oct 31 13:00:02.004497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.004472 5074 generic.go:356] "Generic (PLEG): container finished" podID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerID="1ed691c6cb06d972fe7a8b0707a31fb03d7d9e3b12441403693c6240bbd2b741" exitCode=60 Oct 31 13:00:02.004652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.004558 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" event={"ID":"47acb20a-2db0-487a-8a57-2871d3ac58d0","Type":"ContainerDied","Data":"1ed691c6cb06d972fe7a8b0707a31fb03d7d9e3b12441403693c6240bbd2b741"} Oct 31 13:00:02.004652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.004606 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" event={"ID":"47acb20a-2db0-487a-8a57-2871d3ac58d0","Type":"ContainerStarted","Data":"66c44e4b2797f49756d836679bfa2bb441070b1373a313f98f899c086bfe48c6"} Oct 31 13:00:02.004821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.004793 5074 scope.go:122] "RemoveContainer" containerID="1ed691c6cb06d972fe7a8b0707a31fb03d7d9e3b12441403693c6240bbd2b741" Oct 31 13:00:02.006559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.006459 5074 generic.go:356] "Generic (PLEG): container finished" podID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerID="42c80623f876a445c2a6e0a1ae5bc23131191dcb1bf3ef0b4c4ad55ff17413e2" exitCode=60 Oct 31 13:00:02.006626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.006586 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" event={"ID":"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1","Type":"ContainerDied","Data":"42c80623f876a445c2a6e0a1ae5bc23131191dcb1bf3ef0b4c4ad55ff17413e2"} Oct 31 13:00:02.006626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.006625 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" event={"ID":"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1","Type":"ContainerStarted","Data":"1dadf3ac1fae4ee3f2ce9ee9d0ccb40afbc1beda83666714301eb18007dfe05d"} Oct 31 13:00:02.006790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.006730 5074 scope.go:122] "RemoveContainer" containerID="42c80623f876a445c2a6e0a1ae5bc23131191dcb1bf3ef0b4c4ad55ff17413e2" Oct 31 13:00:02.021465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.021421 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" podStartSLOduration=2.021409768 podStartE2EDuration="2.021409768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:02.017887541 +0000 UTC m=+1696.195605523" watchObservedRunningTime="2026-10-31 13:00:02.021409768 +0000 UTC m=+1696.199127750" Oct 31 13:00:02.040726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:02.040681 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" podStartSLOduration=2.040668756 podStartE2EDuration="2.040668756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:02.036787466 +0000 UTC m=+1696.214505453" watchObservedRunningTime="2026-10-31 13:00:02.040668756 +0000 UTC m=+1696.218386739" Oct 31 13:00:03.020105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.020066 5074 generic.go:356] "Generic (PLEG): container finished" podID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerID="636ddb2a30a85c5c4248b21f5030d4ecec937dd7dc13c79db4224a07215fb11f" exitCode=60 Oct 31 13:00:03.020518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.020150 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerDied","Data":"636ddb2a30a85c5c4248b21f5030d4ecec937dd7dc13c79db4224a07215fb11f"} Oct 31 13:00:03.020518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.020477 5074 scope.go:122] "RemoveContainer" containerID="636ddb2a30a85c5c4248b21f5030d4ecec937dd7dc13c79db4224a07215fb11f" Oct 31 13:00:03.022211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.022186 5074 generic.go:356] "Generic (PLEG): container finished" podID="41153fe2-e960-4866-a7f3-737ec64470d8" containerID="2a9106fc1b3a835140569783f0b48737d5873a011eae5812943db02e9d72a446" exitCode=60 Oct 31 13:00:03.022326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.022265 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" event={"ID":"41153fe2-e960-4866-a7f3-737ec64470d8","Type":"ContainerDied","Data":"2a9106fc1b3a835140569783f0b48737d5873a011eae5812943db02e9d72a446"} Oct 31 13:00:03.022481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.022464 5074 scope.go:122] "RemoveContainer" containerID="2a9106fc1b3a835140569783f0b48737d5873a011eae5812943db02e9d72a446" Oct 31 13:00:03.024580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.024558 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" event={"ID":"47acb20a-2db0-487a-8a57-2871d3ac58d0","Type":"ContainerStarted","Data":"eec1aa760e647817510992b82ad83230cf360ad1e9b3308181dda5f95147f885"} Oct 31 13:00:03.026610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.026587 5074 generic.go:356] "Generic (PLEG): container finished" podID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerID="d6a9d0025be8e5a5852887085897b9e6a3f83ad7fd2d5971677d2b7409ec8125" exitCode=60 Oct 31 13:00:03.026727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.026608 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" event={"ID":"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1","Type":"ContainerDied","Data":"d6a9d0025be8e5a5852887085897b9e6a3f83ad7fd2d5971677d2b7409ec8125"} Oct 31 13:00:03.026727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.026716 5074 scope.go:122] "RemoveContainer" containerID="42c80623f876a445c2a6e0a1ae5bc23131191dcb1bf3ef0b4c4ad55ff17413e2" Oct 31 13:00:03.026939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.026923 5074 scope.go:122] "RemoveContainer" containerID="d6a9d0025be8e5a5852887085897b9e6a3f83ad7fd2d5971677d2b7409ec8125" Oct 31 13:00:03.027149 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:03.027129 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55_odh-ai-gateway-infra(14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" Oct 31 13:00:03.067722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:03.067680 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" podStartSLOduration=3.067666246 podStartE2EDuration="3.067666246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:03.064485591 +0000 UTC m=+1697.242203596" watchObservedRunningTime="2026-10-31 13:00:03.067666246 +0000 UTC m=+1697.245384228" Oct 31 13:00:04.037375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.037340 5074 generic.go:356] "Generic (PLEG): container finished" podID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerID="eec1aa760e647817510992b82ad83230cf360ad1e9b3308181dda5f95147f885" exitCode=60 Oct 31 13:00:04.037850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.037418 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" event={"ID":"47acb20a-2db0-487a-8a57-2871d3ac58d0","Type":"ContainerDied","Data":"eec1aa760e647817510992b82ad83230cf360ad1e9b3308181dda5f95147f885"} Oct 31 13:00:04.037850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.037474 5074 scope.go:122] "RemoveContainer" containerID="1ed691c6cb06d972fe7a8b0707a31fb03d7d9e3b12441403693c6240bbd2b741" Oct 31 13:00:04.037850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.037845 5074 scope.go:122] "RemoveContainer" containerID="eec1aa760e647817510992b82ad83230cf360ad1e9b3308181dda5f95147f885" Oct 31 13:00:04.038326 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:04.038275 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c_odh-ai-gateway-infra(47acb20a-2db0-487a-8a57-2871d3ac58d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" Oct 31 13:00:04.039817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.039798 5074 scope.go:122] "RemoveContainer" containerID="d6a9d0025be8e5a5852887085897b9e6a3f83ad7fd2d5971677d2b7409ec8125" Oct 31 13:00:04.040169 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:04.040146 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55_odh-ai-gateway-infra(14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" Oct 31 13:00:04.041599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.041566 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerStarted","Data":"a866bdc5d26be3eb365d888542e1916027eb2b7ac72f811e22d284916704d36a"} Oct 31 13:00:04.043692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.043670 5074 generic.go:356] "Generic (PLEG): container finished" podID="41153fe2-e960-4866-a7f3-737ec64470d8" containerID="0aa1484504ee84bac07babd71152f9866a406577c0a1d16365dce37f44361295" exitCode=60 Oct 31 13:00:04.043800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.043719 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" event={"ID":"41153fe2-e960-4866-a7f3-737ec64470d8","Type":"ContainerDied","Data":"0aa1484504ee84bac07babd71152f9866a406577c0a1d16365dce37f44361295"} Oct 31 13:00:04.044066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.044045 5074 scope.go:122] "RemoveContainer" containerID="0aa1484504ee84bac07babd71152f9866a406577c0a1d16365dce37f44361295" Oct 31 13:00:04.044364 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:04.044344 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7_odh-ai-gateway-infra(41153fe2-e960-4866-a7f3-737ec64470d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" Oct 31 13:00:04.057324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:04.057306 5074 scope.go:122] "RemoveContainer" containerID="2a9106fc1b3a835140569783f0b48737d5873a011eae5812943db02e9d72a446" Oct 31 13:00:05.056060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:05.056028 5074 generic.go:356] "Generic (PLEG): container finished" podID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerID="a866bdc5d26be3eb365d888542e1916027eb2b7ac72f811e22d284916704d36a" exitCode=60 Oct 31 13:00:05.056464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:05.056200 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerDied","Data":"a866bdc5d26be3eb365d888542e1916027eb2b7ac72f811e22d284916704d36a"} Oct 31 13:00:05.056464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:05.056246 5074 scope.go:122] "RemoveContainer" containerID="636ddb2a30a85c5c4248b21f5030d4ecec937dd7dc13c79db4224a07215fb11f" Oct 31 13:00:05.056609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:05.056591 5074 scope.go:122] "RemoveContainer" containerID="a866bdc5d26be3eb365d888542e1916027eb2b7ac72f811e22d284916704d36a" Oct 31 13:00:05.056992 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:05.056968 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5_odh-ai-gateway-infra(fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" Oct 31 13:00:05.059079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:05.059058 5074 scope.go:122] "RemoveContainer" containerID="0aa1484504ee84bac07babd71152f9866a406577c0a1d16365dce37f44361295" Oct 31 13:00:05.059452 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:05.059430 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7_odh-ai-gateway-infra(41153fe2-e960-4866-a7f3-737ec64470d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" Oct 31 13:00:05.060927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:05.060909 5074 scope.go:122] "RemoveContainer" containerID="eec1aa760e647817510992b82ad83230cf360ad1e9b3308181dda5f95147f885" Oct 31 13:00:05.061224 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:05.061204 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c_odh-ai-gateway-infra(47acb20a-2db0-487a-8a57-2871d3ac58d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" Oct 31 13:00:06.072007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:06.071973 5074 scope.go:122] "RemoveContainer" containerID="a866bdc5d26be3eb365d888542e1916027eb2b7ac72f811e22d284916704d36a" Oct 31 13:00:06.072511 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:06.072370 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5_odh-ai-gateway-infra(fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" Oct 31 13:00:07.135654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:07.135438 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:00:13.941406 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:13.934218 5074 scope.go:122] "RemoveContainer" containerID="d6a9d0025be8e5a5852887085897b9e6a3f83ad7fd2d5971677d2b7409ec8125" Oct 31 13:00:13.941406 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:13.935573 5074 scope.go:122] "RemoveContainer" containerID="eec1aa760e647817510992b82ad83230cf360ad1e9b3308181dda5f95147f885" Oct 31 13:00:15.225385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.225303 5074 generic.go:356] "Generic (PLEG): container finished" podID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerID="63810a44a2610d4b8c60de7a6ad6ebff4e983f549ef0dd64f4b0823672d56f01" exitCode=60 Oct 31 13:00:15.225835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.225377 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" event={"ID":"47acb20a-2db0-487a-8a57-2871d3ac58d0","Type":"ContainerDied","Data":"63810a44a2610d4b8c60de7a6ad6ebff4e983f549ef0dd64f4b0823672d56f01"} Oct 31 13:00:15.225835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.225491 5074 scope.go:122] "RemoveContainer" containerID="eec1aa760e647817510992b82ad83230cf360ad1e9b3308181dda5f95147f885" Oct 31 13:00:15.225949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.225930 5074 scope.go:122] "RemoveContainer" containerID="63810a44a2610d4b8c60de7a6ad6ebff4e983f549ef0dd64f4b0823672d56f01" Oct 31 13:00:15.226333 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:15.226285 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c_odh-ai-gateway-infra(47acb20a-2db0-487a-8a57-2871d3ac58d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" Oct 31 13:00:15.227582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.227561 5074 generic.go:356] "Generic (PLEG): container finished" podID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerID="c8ae2c020b668a5a27618724b4891f6cf39a5702228f6407eff44639dd1b8e15" exitCode=60 Oct 31 13:00:15.227714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.227660 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" event={"ID":"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1","Type":"ContainerDied","Data":"c8ae2c020b668a5a27618724b4891f6cf39a5702228f6407eff44639dd1b8e15"} Oct 31 13:00:15.228055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.228040 5074 scope.go:122] "RemoveContainer" containerID="c8ae2c020b668a5a27618724b4891f6cf39a5702228f6407eff44639dd1b8e15" Oct 31 13:00:15.228350 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:15.228332 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55_odh-ai-gateway-infra(14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" Oct 31 13:00:15.241492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:15.241472 5074 scope.go:122] "RemoveContainer" containerID="d6a9d0025be8e5a5852887085897b9e6a3f83ad7fd2d5971677d2b7409ec8125" Oct 31 13:00:16.009661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.009113 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7"] Oct 31 13:00:16.359013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.348033 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c"] Oct 31 13:00:16.392711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.392678 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55"] Oct 31 13:00:16.404834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.404804 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:16.499654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.498873 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41153fe2-e960-4866-a7f3-737ec64470d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41153fe2-e960-4866-a7f3-737ec64470d8-maas-api-service-ca\") pod \"41153fe2-e960-4866-a7f3-737ec64470d8\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " Oct 31 13:00:16.499654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.499143 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41153fe2-e960-4866-a7f3-737ec64470d8-kube-api-access-frdpn\" (UniqueName: \"kubernetes.io/projected/41153fe2-e960-4866-a7f3-737ec64470d8-kube-api-access-frdpn\") pod \"41153fe2-e960-4866-a7f3-737ec64470d8\" (UID: \"41153fe2-e960-4866-a7f3-737ec64470d8\") " Oct 31 13:00:16.499654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.499560 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41153fe2-e960-4866-a7f3-737ec64470d8-maas-api-service-ca" pod "41153fe2-e960-4866-a7f3-737ec64470d8" (UID: "41153fe2-e960-4866-a7f3-737ec64470d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:16.523963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.502234 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41153fe2-e960-4866-a7f3-737ec64470d8-kube-api-access-frdpn" pod "41153fe2-e960-4866-a7f3-737ec64470d8" (UID: "41153fe2-e960-4866-a7f3-737ec64470d8"). InnerVolumeSpecName "kube-api-access-frdpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:16.604597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.604512 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdpn\" (UniqueName: \"kubernetes.io/projected/41153fe2-e960-4866-a7f3-737ec64470d8-kube-api-access-frdpn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:16.604597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.604540 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41153fe2-e960-4866-a7f3-737ec64470d8-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:16.739113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.739088 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:16.810812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.810785 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:16.908072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.907994 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-kube-api-access-xfbm6\" (UniqueName: \"kubernetes.io/projected/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-kube-api-access-xfbm6\") pod \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " Oct 31 13:00:16.908259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.908118 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-maas-api-service-ca\") pod \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\" (UID: \"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1\") " Oct 31 13:00:16.909045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.908802 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-maas-api-service-ca" pod "14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" (UID: "14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:16.909045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.909014 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:16.911549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:16.911510 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-kube-api-access-xfbm6" pod "14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" (UID: "14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1"). InnerVolumeSpecName "kube-api-access-xfbm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:17.009723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.009690 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47acb20a-2db0-487a-8a57-2871d3ac58d0-kube-api-access-gjjjl\" (UniqueName: \"kubernetes.io/projected/47acb20a-2db0-487a-8a57-2871d3ac58d0-kube-api-access-gjjjl\") pod \"47acb20a-2db0-487a-8a57-2871d3ac58d0\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " Oct 31 13:00:17.009918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.009765 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47acb20a-2db0-487a-8a57-2871d3ac58d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47acb20a-2db0-487a-8a57-2871d3ac58d0-maas-api-service-ca\") pod \"47acb20a-2db0-487a-8a57-2871d3ac58d0\" (UID: \"47acb20a-2db0-487a-8a57-2871d3ac58d0\") " Oct 31 13:00:17.010241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.010213 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47acb20a-2db0-487a-8a57-2871d3ac58d0-maas-api-service-ca" pod "47acb20a-2db0-487a-8a57-2871d3ac58d0" (UID: "47acb20a-2db0-487a-8a57-2871d3ac58d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:17.010443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.010423 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfbm6\" (UniqueName: \"kubernetes.io/projected/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1-kube-api-access-xfbm6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:17.010542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.010452 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47acb20a-2db0-487a-8a57-2871d3ac58d0-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:17.012754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.012730 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47acb20a-2db0-487a-8a57-2871d3ac58d0-kube-api-access-gjjjl" pod "47acb20a-2db0-487a-8a57-2871d3ac58d0" (UID: "47acb20a-2db0-487a-8a57-2871d3ac58d0"). InnerVolumeSpecName "kube-api-access-gjjjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:17.112846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.112809 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjjl\" (UniqueName: \"kubernetes.io/projected/47acb20a-2db0-487a-8a57-2871d3ac58d0-kube-api-access-gjjjl\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:17.371484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.371452 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" Oct 31 13:00:17.371970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.371493 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7" event={"ID":"41153fe2-e960-4866-a7f3-737ec64470d8","Type":"ContainerDied","Data":"8b36090ec84876f865c167fd1111cdc003baef2850f6d564292d8e728d08f30e"} Oct 31 13:00:17.371970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.371574 5074 scope.go:122] "RemoveContainer" containerID="0aa1484504ee84bac07babd71152f9866a406577c0a1d16365dce37f44361295" Oct 31 13:00:17.373489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.373462 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" Oct 31 13:00:17.373618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.373488 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c" event={"ID":"47acb20a-2db0-487a-8a57-2871d3ac58d0","Type":"ContainerDied","Data":"66c44e4b2797f49756d836679bfa2bb441070b1373a313f98f899c086bfe48c6"} Oct 31 13:00:17.375184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.375160 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" event={"ID":"14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1","Type":"ContainerDied","Data":"1dadf3ac1fae4ee3f2ce9ee9d0ccb40afbc1beda83666714301eb18007dfe05d"} Oct 31 13:00:17.375362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.375186 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55" Oct 31 13:00:17.390673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.390602 5074 scope.go:122] "RemoveContainer" containerID="63810a44a2610d4b8c60de7a6ad6ebff4e983f549ef0dd64f4b0823672d56f01" Oct 31 13:00:17.470111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.470075 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c"] Oct 31 13:00:17.472926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.472901 5074 scope.go:122] "RemoveContainer" containerID="c8ae2c020b668a5a27618724b4891f6cf39a5702228f6407eff44639dd1b8e15" Oct 31 13:00:17.478347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.478305 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890860-6494c"] Oct 31 13:00:17.485388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.485370 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7"] Oct 31 13:00:17.494764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.494738 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c4167730-29890860-hxdk7"] Oct 31 13:00:17.499592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.499565 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55"] Oct 31 13:00:17.504311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.504283 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890860-c8h55"] Oct 31 13:00:17.932536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.932504 5074 scope.go:122] "RemoveContainer" containerID="a866bdc5d26be3eb365d888542e1916027eb2b7ac72f811e22d284916704d36a" Oct 31 13:00:17.943086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.943057 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" path="/var/lib/kubelet/pods/14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1/volumes" Oct 31 13:00:17.943476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.943460 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" path="/var/lib/kubelet/pods/41153fe2-e960-4866-a7f3-737ec64470d8/volumes" Oct 31 13:00:17.943889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:17.943873 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" path="/var/lib/kubelet/pods/47acb20a-2db0-487a-8a57-2871d3ac58d0/volumes" Oct 31 13:00:18.388626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:18.388582 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerStarted","Data":"f57e0401d0fe2b8d566f06f348d4ffad15b43878c02b36b2596183b029cd5b5c"} Oct 31 13:00:19.403836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:19.403793 5074 generic.go:356] "Generic (PLEG): container finished" podID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerID="f57e0401d0fe2b8d566f06f348d4ffad15b43878c02b36b2596183b029cd5b5c" exitCode=60 Oct 31 13:00:19.404303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:19.403852 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerDied","Data":"f57e0401d0fe2b8d566f06f348d4ffad15b43878c02b36b2596183b029cd5b5c"} Oct 31 13:00:19.404303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:19.403931 5074 scope.go:122] "RemoveContainer" containerID="a866bdc5d26be3eb365d888542e1916027eb2b7ac72f811e22d284916704d36a" Oct 31 13:00:19.404438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:19.404419 5074 scope.go:122] "RemoveContainer" containerID="f57e0401d0fe2b8d566f06f348d4ffad15b43878c02b36b2596183b029cd5b5c" Oct 31 13:00:19.404847 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:19.404825 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5_odh-ai-gateway-infra(fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" Oct 31 13:00:19.425835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:19.425807 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5"] Oct 31 13:00:20.578739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:20.578714 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:20.707705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:20.705380 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-maas-api-service-ca\") pod \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " Oct 31 13:00:20.707705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:20.705474 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-kube-api-access-mdbgn\" (UniqueName: \"kubernetes.io/projected/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-kube-api-access-mdbgn\") pod \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\" (UID: \"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f\") " Oct 31 13:00:20.707705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:20.707423 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-maas-api-service-ca" pod "fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" (UID: "fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:20.713049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:20.712983 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-kube-api-access-mdbgn" pod "fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" (UID: "fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f"). InnerVolumeSpecName "kube-api-access-mdbgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:20.808665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:20.808569 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:20.808665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:20.808607 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdbgn\" (UniqueName: \"kubernetes.io/projected/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f-kube-api-access-mdbgn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:21.433918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:21.433885 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" Oct 31 13:00:21.433918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:21.433893 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5" event={"ID":"fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f","Type":"ContainerDied","Data":"c8b08082af3d5b5c1e67bf268923572cf110a3c438d3d177f21a6b73292dac38"} Oct 31 13:00:21.434251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:21.433966 5074 scope.go:122] "RemoveContainer" containerID="f57e0401d0fe2b8d566f06f348d4ffad15b43878c02b36b2596183b029cd5b5c" Oct 31 13:00:21.466650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:21.466609 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5"] Oct 31 13:00:21.475104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:21.475068 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-107955-29890860-6p4c5"] Oct 31 13:00:21.940456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:21.940417 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" path="/var/lib/kubelet/pods/fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f/volumes" Oct 31 13:00:24.021675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.021620 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5"] Oct 31 13:00:24.022260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022243 5074 cpu_manager.go:414] "containerMap: removing container" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022261 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022277 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.022303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022283 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.022303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022298 5074 cpu_manager.go:414] "containerMap: removing container" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022303 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022317 5074 cpu_manager.go:414] "containerMap: removing container" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022322 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022334 5074 cpu_manager.go:414] "containerMap: removing container" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022339 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022347 5074 cpu_manager.go:414] "containerMap: removing container" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022352 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022357 5074 cpu_manager.go:414] "containerMap: removing container" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022362 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022369 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022373 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022384 5074 cpu_manager.go:414] "containerMap: removing container" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022388 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022394 5074 cpu_manager.go:414] "containerMap: removing container" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022398 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022407 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.022459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022411 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022506 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022516 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022522 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022527 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022534 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022539 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b65a0a-934e-4fc6-8a3a-8d467d3fd8f1" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022547 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:24.022943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.022553 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:24.027712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.027690 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.034013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.033890 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:00:24.034013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.033980 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:00:24.035339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.035312 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:00:24.035449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.035400 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:00:24.038662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.038610 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5"] Oct 31 13:00:24.049557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.049527 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.049759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.049734 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.049853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.049772 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqz2\" (UniqueName: \"kubernetes.io/projected/05a9e022-f6ac-4689-8124-e00b722ebd3f-kube-api-access-ftqz2\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.151245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.151202 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.151434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.151372 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.151434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.151396 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqz2\" (UniqueName: \"kubernetes.io/projected/05a9e022-f6ac-4689-8124-e00b722ebd3f-kube-api-access-ftqz2\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.152478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.152450 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.155499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.155461 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.164066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.164042 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqz2\" (UniqueName: \"kubernetes.io/projected/05a9e022-f6ac-4689-8124-e00b722ebd3f-kube-api-access-ftqz2\") pod \"maas-api-revoke-sub-8e827e56-66kw5\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.352432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.352329 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:24.766379 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:24.766343 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a9e022_f6ac_4689_8124_e00b722ebd3f.slice/crio-c3a962f62ad20fd129331a195de28bf366a04330d225a90d671fd80a09c8ddab WatchSource:0}: Error finding container c3a962f62ad20fd129331a195de28bf366a04330d225a90d671fd80a09c8ddab: Status 404 returned error can't find the container with id c3a962f62ad20fd129331a195de28bf366a04330d225a90d671fd80a09c8ddab Oct 31 13:00:24.772823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:24.767696 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5"] Oct 31 13:00:25.474979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:25.474941 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" event={"ID":"05a9e022-f6ac-4689-8124-e00b722ebd3f","Type":"ContainerStarted","Data":"9423210e6000d0eb2ac773ac6e34f88141717f1f6022c6a23815cf1f461aa800"} Oct 31 13:00:25.475487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:25.475008 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" event={"ID":"05a9e022-f6ac-4689-8124-e00b722ebd3f","Type":"ContainerStarted","Data":"c3a962f62ad20fd129331a195de28bf366a04330d225a90d671fd80a09c8ddab"} Oct 31 13:00:25.506466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:25.506400 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" podStartSLOduration=2.5063803719999997 podStartE2EDuration="2.506380372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:25.500365312 +0000 UTC m=+1719.678083296" watchObservedRunningTime="2026-10-31 13:00:25.506380372 +0000 UTC m=+1719.684098359" Oct 31 13:00:25.992966 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:25.992244 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:00:26.485670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:26.485624 5074 generic.go:356] "Generic (PLEG): container finished" podID="05a9e022-f6ac-4689-8124-e00b722ebd3f" containerID="9423210e6000d0eb2ac773ac6e34f88141717f1f6022c6a23815cf1f461aa800" exitCode=0 Oct 31 13:00:26.485670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:26.485664 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" event={"ID":"05a9e022-f6ac-4689-8124-e00b722ebd3f","Type":"ContainerDied","Data":"9423210e6000d0eb2ac773ac6e34f88141717f1f6022c6a23815cf1f461aa800"} Oct 31 13:00:27.817407 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.817376 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:27.891311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.891276 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-xks-ca\") pod \"05a9e022-f6ac-4689-8124-e00b722ebd3f\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " Oct 31 13:00:27.893519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.893489 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a9e022-f6ac-4689-8124-e00b722ebd3f-kube-api-access-ftqz2\" (UniqueName: \"kubernetes.io/projected/05a9e022-f6ac-4689-8124-e00b722ebd3f-kube-api-access-ftqz2\") pod \"05a9e022-f6ac-4689-8124-e00b722ebd3f\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " Oct 31 13:00:27.894110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.893725 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-openshift-ca\") pod \"05a9e022-f6ac-4689-8124-e00b722ebd3f\" (UID: \"05a9e022-f6ac-4689-8124-e00b722ebd3f\") " Oct 31 13:00:27.894369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.894070 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-openshift-ca" pod "05a9e022-f6ac-4689-8124-e00b722ebd3f" (UID: "05a9e022-f6ac-4689-8124-e00b722ebd3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:00:27.895451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.895407 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:27.899720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.899621 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a9e022-f6ac-4689-8124-e00b722ebd3f-kube-api-access-ftqz2" pod "05a9e022-f6ac-4689-8124-e00b722ebd3f" (UID: "05a9e022-f6ac-4689-8124-e00b722ebd3f"). InnerVolumeSpecName "kube-api-access-ftqz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:00:27.903674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.903574 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-xks-ca" pod "05a9e022-f6ac-4689-8124-e00b722ebd3f" (UID: "05a9e022-f6ac-4689-8124-e00b722ebd3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:00:27.997470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.997435 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftqz2\" (UniqueName: \"kubernetes.io/projected/05a9e022-f6ac-4689-8124-e00b722ebd3f-kube-api-access-ftqz2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:27.997470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:27.997467 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a9e022-f6ac-4689-8124-e00b722ebd3f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:00:28.506375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:28.506339 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" Oct 31 13:00:28.506573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:28.506368 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5" event={"ID":"05a9e022-f6ac-4689-8124-e00b722ebd3f","Type":"ContainerDied","Data":"c3a962f62ad20fd129331a195de28bf366a04330d225a90d671fd80a09c8ddab"} Oct 31 13:00:28.506573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:28.506420 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3a962f62ad20fd129331a195de28bf366a04330d225a90d671fd80a09c8ddab" Oct 31 13:00:36.040020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:36.039979 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658"] Oct 31 13:00:36.044081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:36.044055 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891c6a76-mf658"] Oct 31 13:00:37.945399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:37.945352 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f8e68d-30f9-46dc-b9e1-3b404add4132" path="/var/lib/kubelet/pods/20f8e68d-30f9-46dc-b9e1-3b404add4132/volumes" Oct 31 13:00:40.922873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.922829 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4"] Oct 31 13:00:40.925788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.925758 5074 cpu_manager.go:414] "containerMap: removing container" podUID="05a9e022-f6ac-4689-8124-e00b722ebd3f" containerName="revoke-keys" Oct 31 13:00:40.925788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.925786 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a9e022-f6ac-4689-8124-e00b722ebd3f" containerName="revoke-keys" Oct 31 13:00:40.926886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.926859 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="41153fe2-e960-4866-a7f3-737ec64470d8" containerName="cleanup" Oct 31 13:00:40.926999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.926895 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="47acb20a-2db0-487a-8a57-2871d3ac58d0" containerName="cleanup" Oct 31 13:00:40.926999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.926919 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a9e022-f6ac-4689-8124-e00b722ebd3f" containerName="revoke-keys" Oct 31 13:00:40.926999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.926946 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb71e1a0-3a9e-4dd0-be4f-736c08d62d5f" containerName="cleanup" Oct 31 13:00:40.938434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.938221 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4"] Oct 31 13:00:40.938698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.938470 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:40.955225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.955199 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-adfd11-dockercfg-t2zxm\"" Oct 31 13:00:41.014658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:40.991546 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-adfd11\"" Oct 31 13:00:41.038542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.038062 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a4d2c6-35c8-40f8-a330-d40e153138d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:41.038904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.038872 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crf4s\" (UniqueName: \"kubernetes.io/projected/45a4d2c6-35c8-40f8-a330-d40e153138d2-kube-api-access-crf4s\") pod \"payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:41.140508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.140472 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crf4s\" (UniqueName: \"kubernetes.io/projected/45a4d2c6-35c8-40f8-a330-d40e153138d2-kube-api-access-crf4s\") pod \"payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:41.140719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.140703 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a4d2c6-35c8-40f8-a330-d40e153138d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:41.141403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.141375 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a4d2c6-35c8-40f8-a330-d40e153138d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:41.172267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.172231 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crf4s\" (UniqueName: \"kubernetes.io/projected/45a4d2c6-35c8-40f8-a330-d40e153138d2-kube-api-access-crf4s\") pod \"payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:41.340326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.340285 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:41.641610 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:41.641578 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a4d2c6_35c8_40f8_a330_d40e153138d2.slice/crio-6a456c656edb8745403bedfa33466d1ad6c7dd896d429e46b77c1e7d38ea6b56 WatchSource:0}: Error finding container 6a456c656edb8745403bedfa33466d1ad6c7dd896d429e46b77c1e7d38ea6b56: Status 404 returned error can't find the container with id 6a456c656edb8745403bedfa33466d1ad6c7dd896d429e46b77c1e7d38ea6b56 Oct 31 13:00:41.642530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:41.642507 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4"] Oct 31 13:00:42.657516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:42.657462 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" event={"ID":"45a4d2c6-35c8-40f8-a330-d40e153138d2","Type":"ContainerStarted","Data":"3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a"} Oct 31 13:00:42.658088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:42.657522 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" event={"ID":"45a4d2c6-35c8-40f8-a330-d40e153138d2","Type":"ContainerStarted","Data":"6a456c656edb8745403bedfa33466d1ad6c7dd896d429e46b77c1e7d38ea6b56"} Oct 31 13:00:42.658088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:42.657711 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:42.683966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:42.683908 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" podStartSLOduration=2.683893687 podStartE2EDuration="2.683893687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:42.677124016 +0000 UTC m=+1736.854842001" watchObservedRunningTime="2026-10-31 13:00:42.683893687 +0000 UTC m=+1736.861611670" Oct 31 13:00:44.463280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.463242 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6"] Oct 31 13:00:44.467546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.467521 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.499091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.497599 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.499440 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.499554 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.499589 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.499765 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.499819 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.499853 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.499887 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.500247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.500015 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjrp\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-kube-api-access-vhjrp\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.512241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.512201 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6"] Oct 31 13:00:44.571376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.571218 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3128e5-openshift-default-dockercfg-chgx7\"" Oct 31 13:00:44.602306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602246 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602323 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602360 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602392 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602509 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjrp\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-kube-api-access-vhjrp\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602556 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602618 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602702 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.602755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.602733 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.603318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.603289 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-credential-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.604224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.604197 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istiod-ca-cert\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.606112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.604938 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-certs\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.606242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.605892 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-socket\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.606336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.606062 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-data\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.631699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.616997 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-podinfo\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.631699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.627552 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-token\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.631699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.628272 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjrp\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-kube-api-access-vhjrp\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.631699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.630867 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-envoy\") pod \"e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:44.844659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:44.838881 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:45.181025 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:45.180978 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a23c87_a538_4ba2_b2fe_90ba1e4d6378.slice/crio-c5b8f79dc4b1fcdfe800a9b8aa65462f4b76bc3f59a56d855250e45ee1f0f91b WatchSource:0}: Error finding container c5b8f79dc4b1fcdfe800a9b8aa65462f4b76bc3f59a56d855250e45ee1f0f91b: Status 404 returned error can't find the container with id c5b8f79dc4b1fcdfe800a9b8aa65462f4b76bc3f59a56d855250e45ee1f0f91b Oct 31 13:00:45.182855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.182824 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6"] Oct 31 13:00:45.183364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.183328 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:45.183445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.183412 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:45.183545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.183456 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:45.727555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.727511 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" event={"ID":"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378","Type":"ContainerStarted","Data":"c3bc704885c38521fbb50aa34bb6f38d0b8d67215a82a3fd177cf19e454542cf"} Oct 31 13:00:45.728220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.728192 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" event={"ID":"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378","Type":"ContainerStarted","Data":"c5b8f79dc4b1fcdfe800a9b8aa65462f4b76bc3f59a56d855250e45ee1f0f91b"} Oct 31 13:00:45.755576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.755528 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podStartSLOduration=1.7555150739999998 podStartE2EDuration="1.755515074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:45.753262607 +0000 UTC m=+1739.930980593" watchObservedRunningTime="2026-10-31 13:00:45.755515074 +0000 UTC m=+1739.933233058" Oct 31 13:00:45.840943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:45.840893 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:46.605123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.605083 5074 scope.go:122] "RemoveContainer" containerID="1f71ea3946aa8fcb1ad554798910e0efe1b2d8ce1bb09df8cbb569682a08e880" Oct 31 13:00:46.616535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.616509 5074 scope.go:122] "RemoveContainer" containerID="395b9906a3cb4019d12ec66091233c1b826a6ae2a87dcaa57f41f77443d0ec85" Oct 31 13:00:46.626743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.626724 5074 scope.go:122] "RemoveContainer" containerID="0157079f3a1605c8e137e67b01db7efd498f4acf279d278bb045f2edf0d1320f" Oct 31 13:00:46.636139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.636119 5074 scope.go:122] "RemoveContainer" containerID="df9a4ec1a1417310a754fca53193b28555d4176bae90ebbcdb6905c765baaeca" Oct 31 13:00:46.645429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.645410 5074 scope.go:122] "RemoveContainer" containerID="8daba1a8dbda807d81ffbb46d5f7f0730b16f9100fadf4558b811941ae50e10e" Oct 31 13:00:46.653959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.653941 5074 scope.go:122] "RemoveContainer" containerID="cfcdc8fd7aba5b309f22774ad447462d509de7fb591e06381273cf04d4094e2d" Oct 31 13:00:46.662859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.662843 5074 scope.go:122] "RemoveContainer" containerID="1e244455942145df08ec43701db703f23de3c7bb8f2afe762442849a8ac6fd3e" Oct 31 13:00:46.676875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.674466 5074 scope.go:122] "RemoveContainer" containerID="ae1bb5c20060d97c78da9912b6d64bf451ce35debf8d17cae4e534a24f6557ea" Oct 31 13:00:46.842058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.842032 5074 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:00:46.842410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:46.842088 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:00:47.839778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:47.839736 5074 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:00:47.839956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:47.839796 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:00:48.384722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.384623 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:00:48.732202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.732161 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47"] Oct 31 13:00:48.738925 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.738892 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:48.744051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.744028 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dc2db4-dockercfg-9t6bh\"" Oct 31 13:00:48.744600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.744577 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dc2db4\"" Oct 31 13:00:48.755959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.755934 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47"] Oct 31 13:00:48.823782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.823737 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:48.823972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.823944 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294v6\" (UniqueName: \"kubernetes.io/projected/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-kube-api-access-294v6\") pod \"payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:48.842661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.842289 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5"] Oct 31 13:00:48.902695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.886840 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:48.913580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.903262 5074 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:00:48.913580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.905303 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:00:48.987719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.979197 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294v6\" (UniqueName: \"kubernetes.io/projected/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-kube-api-access-294v6\") pod \"payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:48.998685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.989182 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85c6\" (UniqueName: \"kubernetes.io/projected/acbed071-cd2b-4ca4-9224-2651ce86449f-kube-api-access-q85c6\") pod \"payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:48.999088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:48.999058 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:49.020397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.020357 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbed071-cd2b-4ca4-9224-2651ce86449f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:49.051716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.016579 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:49.051942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.038160 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5"] Oct 31 13:00:49.062968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.040761 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-294v6\" (UniqueName: \"kubernetes.io/projected/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-kube-api-access-294v6\") pod \"payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:49.125843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.125551 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbed071-cd2b-4ca4-9224-2651ce86449f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:49.129444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.129414 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85c6\" (UniqueName: \"kubernetes.io/projected/acbed071-cd2b-4ca4-9224-2651ce86449f-kube-api-access-q85c6\") pod \"payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:49.130645 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.126886 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbed071-cd2b-4ca4-9224-2651ce86449f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:49.163801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.163768 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85c6\" (UniqueName: \"kubernetes.io/projected/acbed071-cd2b-4ca4-9224-2651ce86449f-kube-api-access-q85c6\") pod \"payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:49.243151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.242998 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:49.358212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.358180 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:49.821129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.821036 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5"] Oct 31 13:00:49.822101 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:49.822059 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbed071_cd2b_4ca4_9224_2651ce86449f.slice/crio-83711368a00e2923fad70fabd4b225859dea717ad9417eb5c304bb6d38c7ecc1 WatchSource:0}: Error finding container 83711368a00e2923fad70fabd4b225859dea717ad9417eb5c304bb6d38c7ecc1: Status 404 returned error can't find the container with id 83711368a00e2923fad70fabd4b225859dea717ad9417eb5c304bb6d38c7ecc1 Oct 31 13:00:49.839968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.839939 5074 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:00:49.840092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.840003 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:00:49.945291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:49.945256 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47"] Oct 31 13:00:50.842272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.842233 5074 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:00:50.842966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.842920 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:00:50.850374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.850337 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" event={"ID":"acbed071-cd2b-4ca4-9224-2651ce86449f","Type":"ContainerStarted","Data":"1f88fd9a09121fb233500e83a0d6125e974598d76a6bf34659646c74d56ce3ae"} Oct 31 13:00:50.850530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.850433 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" event={"ID":"acbed071-cd2b-4ca4-9224-2651ce86449f","Type":"ContainerStarted","Data":"83711368a00e2923fad70fabd4b225859dea717ad9417eb5c304bb6d38c7ecc1"} Oct 31 13:00:50.850530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.850485 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:00:50.852521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.852497 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" event={"ID":"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c","Type":"ContainerStarted","Data":"84a69615f760cfeecdc8e08be508de075808f4959b19e424d9701e52098d78f9"} Oct 31 13:00:50.852713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.852696 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" event={"ID":"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c","Type":"ContainerStarted","Data":"888b9f371ea2c18d896039b7324406228ec836704572c6ae65a198d50f333f4b"} Oct 31 13:00:50.853470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.853376 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:00:50.911842 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.911778 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" podStartSLOduration=2.908002815 podStartE2EDuration="2.908002815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:50.900312909 +0000 UTC m=+1745.078030895" watchObservedRunningTime="2026-10-31 13:00:50.908002815 +0000 UTC m=+1745.085720820" Oct 31 13:00:50.931173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:50.931091 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" podStartSLOduration=2.931071526 podStartE2EDuration="2.931071526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:50.9243634 +0000 UTC m=+1745.102081387" watchObservedRunningTime="2026-10-31 13:00:50.931071526 +0000 UTC m=+1745.108789517" Oct 31 13:00:51.849568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:51.849533 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:51.852317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:51.852291 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:51.877925 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:51.877890 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:00:52.067426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:52.067376 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95"] Oct 31 13:00:52.075713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:52.075682 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq"] Oct 31 13:00:52.075947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:52.075934 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc36eff-w7z95"] Oct 31 13:00:52.083102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:52.083069 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1739a7d6-w8blq"] Oct 31 13:00:53.673112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:53.673078 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:00:53.939143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:53.939066 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d38372e-8ddb-4468-9c91-872546af224c" path="/var/lib/kubelet/pods/1d38372e-8ddb-4468-9c91-872546af224c/volumes" Oct 31 13:00:53.939569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:53.939554 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a88d7a-ce6b-49aa-9be2-014e0df588c6" path="/var/lib/kubelet/pods/58a88d7a-ce6b-49aa-9be2-014e0df588c6/volumes" Oct 31 13:00:55.992654 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:00:55.991690 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:00:56.823023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.822988 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w"] Oct 31 13:00:56.829368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.829343 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.848874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.848844 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-48dcc1-openshift-default-dockercfg-wg7qk\"" Oct 31 13:00:56.849803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.849765 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w"] Oct 31 13:00:56.950087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950055 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-data\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950090 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-envoy\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950203 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-credential-socket\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950262 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-socket\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950335 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3221e591-a5be-469e-85db-5f756c6de8ec-istiod-ca-cert\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950357 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-istio-token\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950399 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2826\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-kube-api-access-s2826\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950559 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3221e591-a5be-469e-85db-5f756c6de8ec-istio-podinfo\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:56.950625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:56.950606 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-certs\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066338 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2826\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-kube-api-access-s2826\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066413 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3221e591-a5be-469e-85db-5f756c6de8ec-istio-podinfo\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066441 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-certs\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066538 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-data\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066566 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-envoy\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066625 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-credential-socket\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066721 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-socket\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066790 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3221e591-a5be-469e-85db-5f756c6de8ec-istiod-ca-cert\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.066824 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-istio-token\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.068366 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-socket\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.068601 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-credential-socket\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.069286 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3221e591-a5be-469e-85db-5f756c6de8ec-istiod-ca-cert\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.069672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.069531 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-certs\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.086978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.073851 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-data\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.151834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.130739 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-istio-token\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.184783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.184747 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2826\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-kube-api-access-s2826\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.203655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.192499 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3221e591-a5be-469e-85db-5f756c6de8ec-istio-podinfo\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.270369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.270335 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-envoy\") pod \"e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.452817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.452725 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:57.759446 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:00:57.759413 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3221e591_a5be_469e_85db_5f756c6de8ec.slice/crio-74e3979c1d6af25454d36ee7d202dd941ca26e6f3a3a62d2dfbc711d3d665f55 WatchSource:0}: Error finding container 74e3979c1d6af25454d36ee7d202dd941ca26e6f3a3a62d2dfbc711d3d665f55: Status 404 returned error can't find the container with id 74e3979c1d6af25454d36ee7d202dd941ca26e6f3a3a62d2dfbc711d3d665f55 Oct 31 13:00:57.759585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.759462 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w"] Oct 31 13:00:57.762555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.762506 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:57.762670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.762599 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:57.762723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.762692 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:00:57.950224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.950189 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" event={"ID":"3221e591-a5be-469e-85db-5f756c6de8ec","Type":"ContainerStarted","Data":"dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2"} Oct 31 13:00:57.950494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.950475 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" event={"ID":"3221e591-a5be-469e-85db-5f756c6de8ec","Type":"ContainerStarted","Data":"74e3979c1d6af25454d36ee7d202dd941ca26e6f3a3a62d2dfbc711d3d665f55"} Oct 31 13:00:57.982264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:57.981715 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podStartSLOduration=1.9816967829999999 podStartE2EDuration="1.981696783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:57.980323079 +0000 UTC m=+1752.158041063" watchObservedRunningTime="2026-10-31 13:00:57.981696783 +0000 UTC m=+1752.159414770" Oct 31 13:00:58.453031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:58.452997 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:58.460305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:58.460278 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:58.956537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:58.956505 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:00:58.957397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:00:58.957374 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:01:01.882937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:01.882908 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:01:02.886798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:02.886768 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:01:03.072883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:03.072844 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r"] Oct 31 13:01:03.081477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:03.081446 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd"] Oct 31 13:01:03.085827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:03.085804 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ee10d77-vkt5r"] Oct 31 13:01:03.089457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:03.089431 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ccdd27-xk7zd"] Oct 31 13:01:03.940836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:03.940802 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d9a82f-53c8-47b0-9f31-bed85e3edf8f" path="/var/lib/kubelet/pods/11d9a82f-53c8-47b0-9f31-bed85e3edf8f/volumes" Oct 31 13:01:03.941353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:03.941338 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5896bbb-8913-4ab9-9c61-563f7f416473" path="/var/lib/kubelet/pods/c5896bbb-8913-4ab9-9c61-563f7f416473/volumes" Oct 31 13:01:04.059541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.059500 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt"] Oct 31 13:01:04.103865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.103835 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.120957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.120927 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-48dcc1\"" Oct 31 13:01:04.136658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.133430 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv"] Oct 31 13:01:04.157524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.157497 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.161749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.161672 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.162025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.162005 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cn8\" (UniqueName: \"kubernetes.io/projected/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-kube-api-access-j2cn8\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.202955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.202924 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt"] Oct 31 13:01:04.211157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.211126 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv"] Oct 31 13:01:04.263381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.263343 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cckc\" (UniqueName: \"kubernetes.io/projected/83eb243b-c95a-4851-943c-d61e2934157f-kube-api-access-4cckc\") pod \"payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.263548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.263394 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.263548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.263468 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cn8\" (UniqueName: \"kubernetes.io/projected/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-kube-api-access-j2cn8\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.263548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.263523 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83eb243b-c95a-4851-943c-d61e2934157f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.264226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.264202 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.271598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.271575 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s"] Oct 31 13:01:04.284546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.284521 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cn8\" (UniqueName: \"kubernetes.io/projected/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-kube-api-access-j2cn8\") pod \"payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.284948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.284927 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c874071-k6l4s"] Oct 31 13:01:04.365525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.365490 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cckc\" (UniqueName: \"kubernetes.io/projected/83eb243b-c95a-4851-943c-d61e2934157f-kube-api-access-4cckc\") pod \"payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.365758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.365589 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83eb243b-c95a-4851-943c-d61e2934157f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.366452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.366429 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83eb243b-c95a-4851-943c-d61e2934157f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.391466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.391435 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cckc\" (UniqueName: \"kubernetes.io/projected/83eb243b-c95a-4851-943c-d61e2934157f-kube-api-access-4cckc\") pod \"payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.503026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.502994 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:04.511573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.511542 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:04.834557 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:01:04.834513 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c3ed7f_16bd_4693_90f4_93bb8b549bcd.slice/crio-7b77c54c6265363817b6c1826e121e6e3573f7273396bdf22442ee8e49c7648d WatchSource:0}: Error finding container 7b77c54c6265363817b6c1826e121e6e3573f7273396bdf22442ee8e49c7648d: Status 404 returned error can't find the container with id 7b77c54c6265363817b6c1826e121e6e3573f7273396bdf22442ee8e49c7648d Oct 31 13:01:04.837274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.836133 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt"] Oct 31 13:01:04.839190 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:01:04.839162 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eb243b_c95a_4851_943c_d61e2934157f.slice/crio-f2ed56a118e71d41c1831ee903cc084e7e20a447e8f50296578bdd29ab74d4cf WatchSource:0}: Error finding container f2ed56a118e71d41c1831ee903cc084e7e20a447e8f50296578bdd29ab74d4cf: Status 404 returned error can't find the container with id f2ed56a118e71d41c1831ee903cc084e7e20a447e8f50296578bdd29ab74d4cf Oct 31 13:01:04.843998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:04.843974 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv"] Oct 31 13:01:05.072282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:05.069583 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" event={"ID":"21c3ed7f-16bd-4693-90f4-93bb8b549bcd","Type":"ContainerStarted","Data":"7b77c54c6265363817b6c1826e121e6e3573f7273396bdf22442ee8e49c7648d"} Oct 31 13:01:05.075704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:05.072940 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" event={"ID":"83eb243b-c95a-4851-943c-d61e2934157f","Type":"ContainerStarted","Data":"f2ed56a118e71d41c1831ee903cc084e7e20a447e8f50296578bdd29ab74d4cf"} Oct 31 13:01:05.943171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:05.943132 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e1d194-61f7-4bc9-8566-1359db135227" path="/var/lib/kubelet/pods/55e1d194-61f7-4bc9-8566-1359db135227/volumes" Oct 31 13:01:06.083152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:06.083105 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" event={"ID":"21c3ed7f-16bd-4693-90f4-93bb8b549bcd","Type":"ContainerStarted","Data":"2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc"} Oct 31 13:01:06.083660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:06.083181 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:06.084898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:06.084871 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" event={"ID":"83eb243b-c95a-4851-943c-d61e2934157f","Type":"ContainerStarted","Data":"0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778"} Oct 31 13:01:06.085021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:06.084931 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:06.149593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:06.149536 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" podStartSLOduration=3.149522982 podStartE2EDuration="3.149522982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:06.119005609 +0000 UTC m=+1760.296723593" watchObservedRunningTime="2026-10-31 13:01:06.149522982 +0000 UTC m=+1760.327240965" Oct 31 13:01:09.834079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:09.834021 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" podStartSLOduration=6.834002884 podStartE2EDuration="6.834002884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:06.149889043 +0000 UTC m=+1760.327607030" watchObservedRunningTime="2026-10-31 13:01:09.834002884 +0000 UTC m=+1764.011720867" Oct 31 13:01:09.836332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:09.836308 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4"] Oct 31 13:01:13.237174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:13.237141 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:01:14.836860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:14.836770 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" podUID="45a4d2c6-35c8-40f8-a330-d40e153138d2" containerName="payload-processing" containerID="cri-o://3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a" gracePeriod=25 Oct 31 13:01:15.238511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.225619 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:01:15.253397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.253361 5074 generic.go:356] "Generic (PLEG): container finished" podID="45a4d2c6-35c8-40f8-a330-d40e153138d2" containerID="3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a" exitCode=0 Oct 31 13:01:15.253737 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.253715 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" event={"ID":"45a4d2c6-35c8-40f8-a330-d40e153138d2","Type":"ContainerDied","Data":"3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a"} Oct 31 13:01:15.253902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.253880 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" event={"ID":"45a4d2c6-35c8-40f8-a330-d40e153138d2","Type":"ContainerDied","Data":"6a456c656edb8745403bedfa33466d1ad6c7dd896d429e46b77c1e7d38ea6b56"} Oct 31 13:01:15.254057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.254041 5074 scope.go:122] "RemoveContainer" containerID="3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a" Oct 31 13:01:15.254560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.254292 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4" Oct 31 13:01:15.274438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.274408 5074 scope.go:122] "RemoveContainer" containerID="3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a" Oct 31 13:01:15.274864 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:01:15.274838 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a\": container with ID starting with 3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a not found: ID does not exist" containerID="3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a" Oct 31 13:01:15.274966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.274880 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a"} err="failed to get container status \"3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a\": rpc error: code = NotFound desc = could not find container \"3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a\": container with ID starting with 3edea87c49a46332aa3b1491a80b399e121f302c40458e4a0de7e5f4c04ffd2a not found: ID does not exist" Oct 31 13:01:15.401269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.399509 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a4d2c6-35c8-40f8-a330-d40e153138d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a4d2c6-35c8-40f8-a330-d40e153138d2-plugins-config-volume\") pod \"45a4d2c6-35c8-40f8-a330-d40e153138d2\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " Oct 31 13:01:15.401269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.399724 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a4d2c6-35c8-40f8-a330-d40e153138d2-kube-api-access-crf4s\" (UniqueName: \"kubernetes.io/projected/45a4d2c6-35c8-40f8-a330-d40e153138d2-kube-api-access-crf4s\") pod \"45a4d2c6-35c8-40f8-a330-d40e153138d2\" (UID: \"45a4d2c6-35c8-40f8-a330-d40e153138d2\") " Oct 31 13:01:15.401269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.401197 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a4d2c6-35c8-40f8-a330-d40e153138d2-plugins-config-volume" pod "45a4d2c6-35c8-40f8-a330-d40e153138d2" (UID: "45a4d2c6-35c8-40f8-a330-d40e153138d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:15.402889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.402857 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a4d2c6-35c8-40f8-a330-d40e153138d2-kube-api-access-crf4s" pod "45a4d2c6-35c8-40f8-a330-d40e153138d2" (UID: "45a4d2c6-35c8-40f8-a330-d40e153138d2"). InnerVolumeSpecName "kube-api-access-crf4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:15.505703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.502918 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crf4s\" (UniqueName: \"kubernetes.io/projected/45a4d2c6-35c8-40f8-a330-d40e153138d2-kube-api-access-crf4s\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:15.505703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.502959 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a4d2c6-35c8-40f8-a330-d40e153138d2-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:15.593118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.593080 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4"] Oct 31 13:01:15.600107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.600076 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-adfd11-78fc7669b7-gh7m4"] Oct 31 13:01:15.939391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:15.939310 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a4d2c6-35c8-40f8-a330-d40e153138d2" path="/var/lib/kubelet/pods/45a4d2c6-35c8-40f8-a330-d40e153138d2/volumes" Oct 31 13:01:16.038404 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:16.038358 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q"] Oct 31 13:01:16.042821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:16.042793 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9511ee0-s4n5q"] Oct 31 13:01:17.124871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:17.124831 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:17.125455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:17.125435 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:17.221110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:17.219941 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566"] Oct 31 13:01:17.238249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:17.238209 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89790774-gd566"] Oct 31 13:01:17.943267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:17.943235 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdb654e-e974-4799-a809-7a51be6baad1" path="/var/lib/kubelet/pods/6cdb654e-e974-4799-a809-7a51be6baad1/volumes" Oct 31 13:01:17.943774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:17.943759 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8366c85-e16e-4973-bcc9-35d2f85c5380" path="/var/lib/kubelet/pods/f8366c85-e16e-4973-bcc9-35d2f85c5380/volumes" Oct 31 13:01:18.113579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:18.113527 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr"] Oct 31 13:01:18.122698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:18.120726 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56318de9-9jmfr"] Oct 31 13:01:18.143264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:18.143230 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6"] Oct 31 13:01:18.143836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:18.143465 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" containerID="cri-o://c3bc704885c38521fbb50aa34bb6f38d0b8d67215a82a3fd177cf19e454542cf" gracePeriod=30 Oct 31 13:01:19.940382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:19.940346 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ff190b-0b3b-4dfe-be15-127856f947ba" path="/var/lib/kubelet/pods/a9ff190b-0b3b-4dfe-be15-127856f947ba/volumes" Oct 31 13:01:21.854847 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:21.854809 5074 patch_prober.go:29] interesting pod/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:01:21.855264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:21.854871 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:01:23.434481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.434452 5074 generic.go:356] "Generic (PLEG): container finished" podID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerID="c3bc704885c38521fbb50aa34bb6f38d0b8d67215a82a3fd177cf19e454542cf" exitCode=0 Oct 31 13:01:23.434862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.434524 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" event={"ID":"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378","Type":"ContainerDied","Data":"c3bc704885c38521fbb50aa34bb6f38d0b8d67215a82a3fd177cf19e454542cf"} Oct 31 13:01:23.434862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.434587 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" event={"ID":"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378","Type":"ContainerDied","Data":"c5b8f79dc4b1fcdfe800a9b8aa65462f4b76bc3f59a56d855250e45ee1f0f91b"} Oct 31 13:01:23.434862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.434604 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5b8f79dc4b1fcdfe800a9b8aa65462f4b76bc3f59a56d855250e45ee1f0f91b" Oct 31 13:01:23.451731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.451697 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:01:23.561956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.561917 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-certs\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.562184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562053 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-token\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.562395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562355 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-certs" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.562558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562538 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-data\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.562694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562661 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istiod-ca-cert\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.562829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562804 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-data" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.562895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562811 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-envoy\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.562954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562899 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-socket\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.563009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.562974 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-podinfo\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.563065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.563019 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-kube-api-access-vhjrp\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-kube-api-access-vhjrp\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.563122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.563072 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-credential-socket\") pod \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\" (UID: \"d5a23c87-a538-4ba2-b2fe-90ba1e4d6378\") " Oct 31 13:01:23.563398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.563374 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istiod-ca-cert" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:23.563620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.563595 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-credential-socket" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.563997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.563971 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.564091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.564000 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.564091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.564015 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.564091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.564031 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.564601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.564574 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-socket" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.566291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.566266 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-podinfo" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:01:23.566434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.566305 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-token" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.566948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.566899 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-kube-api-access-vhjrp" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "kube-api-access-vhjrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:23.566948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.566900 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-envoy" pod "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" (UID: "d5a23c87-a538-4ba2-b2fe-90ba1e4d6378"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:01:23.665109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.665066 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjrp\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-kube-api-access-vhjrp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.665109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.665107 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.665321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.665122 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.665321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.665132 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:23.665321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:23.665141 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:24.444973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:24.444888 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6" Oct 31 13:01:24.516167 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:24.515858 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6"] Oct 31 13:01:24.520394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:24.520361 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3128e5-openshift-default-67b6dbdd5-wj9w6"] Oct 31 13:01:25.941019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:25.940982 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" path="/var/lib/kubelet/pods/d5a23c87-a538-4ba2-b2fe-90ba1e4d6378/volumes" Oct 31 13:01:25.986794 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:01:25.986755 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:01:29.903796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.903758 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj"] Oct 31 13:01:29.904737 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.904716 5074 cpu_manager.go:414] "containerMap: removing container" podUID="45a4d2c6-35c8-40f8-a330-d40e153138d2" containerName="payload-processing" Oct 31 13:01:29.904832 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.904737 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a4d2c6-35c8-40f8-a330-d40e153138d2" containerName="payload-processing" Oct 31 13:01:29.904832 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.904796 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" Oct 31 13:01:29.904832 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.904805 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" Oct 31 13:01:29.904991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.904981 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a4d2c6-35c8-40f8-a330-d40e153138d2" containerName="payload-processing" Oct 31 13:01:29.905053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.904999 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a23c87-a538-4ba2-b2fe-90ba1e4d6378" containerName="istio-proxy" Oct 31 13:01:29.910849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.910826 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:29.920557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.920526 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj"] Oct 31 13:01:29.931768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.931740 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-01e8d2\"" Oct 31 13:01:29.953081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.953050 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj"] Oct 31 13:01:29.957518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.957496 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:29.967219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.967188 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj"] Oct 31 13:01:29.979623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:29.973029 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-01e8d2-dockercfg-wb9jq\"" Oct 31 13:01:30.016381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.016334 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8daef-e8f4-4fac-a139-a9a557b365a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:30.016666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.016624 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gksx\" (UniqueName: \"kubernetes.io/projected/9ed8daef-e8f4-4fac-a139-a9a557b365a5-kube-api-access-9gksx\") pod \"payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:30.132659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.124544 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:30.132659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.124718 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8daef-e8f4-4fac-a139-a9a557b365a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:30.132659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.124884 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q64d\" (UniqueName: \"kubernetes.io/projected/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-kube-api-access-8q64d\") pod \"payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:30.132659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.124917 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gksx\" (UniqueName: \"kubernetes.io/projected/9ed8daef-e8f4-4fac-a139-a9a557b365a5-kube-api-access-9gksx\") pod \"payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:30.132659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.126313 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8daef-e8f4-4fac-a139-a9a557b365a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:30.180416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.180341 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gksx\" (UniqueName: \"kubernetes.io/projected/9ed8daef-e8f4-4fac-a139-a9a557b365a5-kube-api-access-9gksx\") pod \"payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:30.238512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.231844 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q64d\" (UniqueName: \"kubernetes.io/projected/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-kube-api-access-8q64d\") pod \"payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:30.238512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.232033 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:30.238512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.233195 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:30.238512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.237792 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:30.253794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.253531 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q64d\" (UniqueName: \"kubernetes.io/projected/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-kube-api-access-8q64d\") pod \"payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:30.285314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.285273 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:30.569683 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:01:30.569652 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed8daef_e8f4_4fac_a139_a9a557b365a5.slice/crio-d7a94b5c4e2472e4cb6e0044eb492b9edf8d5d7733dd9d8b5e2fcee1549fe06f WatchSource:0}: Error finding container d7a94b5c4e2472e4cb6e0044eb492b9edf8d5d7733dd9d8b5e2fcee1549fe06f: Status 404 returned error can't find the container with id d7a94b5c4e2472e4cb6e0044eb492b9edf8d5d7733dd9d8b5e2fcee1549fe06f Oct 31 13:01:30.569683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.569665 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj"] Oct 31 13:01:30.590964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.590930 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" event={"ID":"9ed8daef-e8f4-4fac-a139-a9a557b365a5","Type":"ContainerStarted","Data":"d7a94b5c4e2472e4cb6e0044eb492b9edf8d5d7733dd9d8b5e2fcee1549fe06f"} Oct 31 13:01:30.605291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:30.605266 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj"] Oct 31 13:01:30.610062 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:01:30.610032 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9ceac4_b56e_4edb_b5a6_36cad784b8f8.slice/crio-30ae9b47b6ba447ab6f939c4534596f624912ab1f303e7e8d090153e2c2e9f78 WatchSource:0}: Error finding container 30ae9b47b6ba447ab6f939c4534596f624912ab1f303e7e8d090153e2c2e9f78: Status 404 returned error can't find the container with id 30ae9b47b6ba447ab6f939c4534596f624912ab1f303e7e8d090153e2c2e9f78 Oct 31 13:01:31.040576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.040533 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw"] Oct 31 13:01:31.046926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.046900 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1457cd2f-xg4hw"] Oct 31 13:01:31.601461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.601422 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" event={"ID":"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8","Type":"ContainerStarted","Data":"e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a"} Oct 31 13:01:31.601683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.601482 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" event={"ID":"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8","Type":"ContainerStarted","Data":"30ae9b47b6ba447ab6f939c4534596f624912ab1f303e7e8d090153e2c2e9f78"} Oct 31 13:01:31.601683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.601524 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:31.603375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.603346 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" event={"ID":"9ed8daef-e8f4-4fac-a139-a9a557b365a5","Type":"ContainerStarted","Data":"e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851"} Oct 31 13:01:31.603485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.603438 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:31.624418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.624377 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" podStartSLOduration=2.624363712 podStartE2EDuration="2.624363712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:31.620373651 +0000 UTC m=+1785.798091634" watchObservedRunningTime="2026-10-31 13:01:31.624363712 +0000 UTC m=+1785.802081696" Oct 31 13:01:31.646741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.646689 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" podStartSLOduration=2.646674367 podStartE2EDuration="2.646674367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:31.642279586 +0000 UTC m=+1785.819997577" watchObservedRunningTime="2026-10-31 13:01:31.646674367 +0000 UTC m=+1785.824392355" Oct 31 13:01:31.941530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:31.941439 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612228c0-e5d3-4efc-ab43-f0b0b29bd5df" path="/var/lib/kubelet/pods/612228c0-e5d3-4efc-ab43-f0b0b29bd5df/volumes" Oct 31 13:01:32.043773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:32.043736 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb"] Oct 31 13:01:32.051395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:32.051353 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6"] Oct 31 13:01:32.056942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:32.056910 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466b7fb5-p8wxb"] Oct 31 13:01:32.067682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:32.063415 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3030e544-28qr6"] Oct 31 13:01:33.942044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:33.942006 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac5cd21-89ad-4d47-ba77-898b4ee4ce59" path="/var/lib/kubelet/pods/cac5cd21-89ad-4d47-ba77-898b4ee4ce59/volumes" Oct 31 13:01:33.942722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:33.942702 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e86576-8ab5-40db-9150-036471eb200b" path="/var/lib/kubelet/pods/e2e86576-8ab5-40db-9150-036471eb200b/volumes" Oct 31 13:01:36.403012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:36.402974 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:01:41.204656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:41.190790 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv"] Oct 31 13:01:41.211786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:41.210849 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt"] Oct 31 13:01:42.625877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:42.625839 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:01:42.628236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:42.628212 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:01:46.207083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.207047 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" podUID="83eb243b-c95a-4851-943c-d61e2934157f" containerName="payload-processing" containerID="cri-o://0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778" gracePeriod=25 Oct 31 13:01:46.211228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.211098 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" podUID="21c3ed7f-16bd-4693-90f4-93bb8b549bcd" containerName="payload-pre-processing" containerID="cri-o://2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc" gracePeriod=25 Oct 31 13:01:46.250932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.250898 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:01:46.252045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.252016 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:01:46.558326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.558291 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:46.651909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.651874 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-plugins-config-volume\") pod \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " Oct 31 13:01:46.652079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.652033 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-kube-api-access-j2cn8\" (UniqueName: \"kubernetes.io/projected/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-kube-api-access-j2cn8\") pod \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\" (UID: \"21c3ed7f-16bd-4693-90f4-93bb8b549bcd\") " Oct 31 13:01:46.652380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.652350 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-plugins-config-volume" pod "21c3ed7f-16bd-4693-90f4-93bb8b549bcd" (UID: "21c3ed7f-16bd-4693-90f4-93bb8b549bcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:46.652573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.652554 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:46.655333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.655298 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-kube-api-access-j2cn8" pod "21c3ed7f-16bd-4693-90f4-93bb8b549bcd" (UID: "21c3ed7f-16bd-4693-90f4-93bb8b549bcd"). InnerVolumeSpecName "kube-api-access-j2cn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:46.664293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.664263 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:46.754112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.754071 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83eb243b-c95a-4851-943c-d61e2934157f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83eb243b-c95a-4851-943c-d61e2934157f-plugins-config-volume\") pod \"83eb243b-c95a-4851-943c-d61e2934157f\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " Oct 31 13:01:46.754312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.754169 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83eb243b-c95a-4851-943c-d61e2934157f-kube-api-access-4cckc\" (UniqueName: \"kubernetes.io/projected/83eb243b-c95a-4851-943c-d61e2934157f-kube-api-access-4cckc\") pod \"83eb243b-c95a-4851-943c-d61e2934157f\" (UID: \"83eb243b-c95a-4851-943c-d61e2934157f\") " Oct 31 13:01:46.754779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.754473 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eb243b-c95a-4851-943c-d61e2934157f-plugins-config-volume" pod "83eb243b-c95a-4851-943c-d61e2934157f" (UID: "83eb243b-c95a-4851-943c-d61e2934157f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:01:46.754779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.754737 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2cn8\" (UniqueName: \"kubernetes.io/projected/21c3ed7f-16bd-4693-90f4-93bb8b549bcd-kube-api-access-j2cn8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:46.754779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.754758 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83eb243b-c95a-4851-943c-d61e2934157f-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:46.757053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.757030 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eb243b-c95a-4851-943c-d61e2934157f-kube-api-access-4cckc" pod "83eb243b-c95a-4851-943c-d61e2934157f" (UID: "83eb243b-c95a-4851-943c-d61e2934157f"). InnerVolumeSpecName "kube-api-access-4cckc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:01:46.824477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.824393 5074 generic.go:356] "Generic (PLEG): container finished" podID="21c3ed7f-16bd-4693-90f4-93bb8b549bcd" containerID="2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc" exitCode=0 Oct 31 13:01:46.824477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.824441 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" Oct 31 13:01:46.824717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.824483 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" event={"ID":"21c3ed7f-16bd-4693-90f4-93bb8b549bcd","Type":"ContainerDied","Data":"2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc"} Oct 31 13:01:46.824717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.824580 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt" event={"ID":"21c3ed7f-16bd-4693-90f4-93bb8b549bcd","Type":"ContainerDied","Data":"7b77c54c6265363817b6c1826e121e6e3573f7273396bdf22442ee8e49c7648d"} Oct 31 13:01:46.824717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.824613 5074 scope.go:122] "RemoveContainer" containerID="2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc" Oct 31 13:01:46.826313 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.826276 5074 generic.go:356] "Generic (PLEG): container finished" podID="83eb243b-c95a-4851-943c-d61e2934157f" containerID="0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778" exitCode=0 Oct 31 13:01:46.826422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.826305 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" event={"ID":"83eb243b-c95a-4851-943c-d61e2934157f","Type":"ContainerDied","Data":"0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778"} Oct 31 13:01:46.826422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.826333 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" Oct 31 13:01:46.826422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.826362 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv" event={"ID":"83eb243b-c95a-4851-943c-d61e2934157f","Type":"ContainerDied","Data":"f2ed56a118e71d41c1831ee903cc084e7e20a447e8f50296578bdd29ab74d4cf"} Oct 31 13:01:46.835438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.835414 5074 scope.go:122] "RemoveContainer" containerID="88c1263411f63fab475af2da7be76ce4b9f48b430626963372dbd715f65cbd49" Oct 31 13:01:46.839351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.839172 5074 scope.go:122] "RemoveContainer" containerID="2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc" Oct 31 13:01:46.839677 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:01:46.839615 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc\": container with ID starting with 2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc not found: ID does not exist" containerID="2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc" Oct 31 13:01:46.839797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.839681 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc"} err="failed to get container status \"2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc\": rpc error: code = NotFound desc = could not find container \"2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc\": container with ID starting with 2bfa1ff2a1df91cbc509c0581a904576b08f3a408a353b72dfbf9ee0bad5cccc not found: ID does not exist" Oct 31 13:01:46.839797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.839706 5074 scope.go:122] "RemoveContainer" containerID="0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778" Oct 31 13:01:46.855879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.855850 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cckc\" (UniqueName: \"kubernetes.io/projected/83eb243b-c95a-4851-943c-d61e2934157f-kube-api-access-4cckc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:01:46.857211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.857192 5074 scope.go:122] "RemoveContainer" containerID="0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778" Oct 31 13:01:46.857527 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:01:46.857503 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778\": container with ID starting with 0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778 not found: ID does not exist" containerID="0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778" Oct 31 13:01:46.857623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.857533 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778"} err="failed to get container status \"0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778\": rpc error: code = NotFound desc = could not find container \"0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778\": container with ID starting with 0bb765c5b97fe0fe74633be3dd24d890aafb3d49d4069bc7941d82dcdd65f778 not found: ID does not exist" Oct 31 13:01:46.875012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.874912 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt"] Oct 31 13:01:46.881664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.881600 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48dcc1-dbd5c47f-xpbrt"] Oct 31 13:01:46.897692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.897663 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv"] Oct 31 13:01:46.903765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:46.903736 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48dcc1-7486d57449-wzlpv"] Oct 31 13:01:47.093360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.093270 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72"] Oct 31 13:01:47.099707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.099673 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7"] Oct 31 13:01:47.111212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.111177 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2db079-msh72"] Oct 31 13:01:47.121331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.121297 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8df54bb-9txv7"] Oct 31 13:01:47.603537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.596679 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7"] Oct 31 13:01:47.603537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.597672 5074 cpu_manager.go:414] "containerMap: removing container" podUID="83eb243b-c95a-4851-943c-d61e2934157f" containerName="payload-processing" Oct 31 13:01:47.603537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.597692 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83eb243b-c95a-4851-943c-d61e2934157f" containerName="payload-processing" Oct 31 13:01:47.603537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.597712 5074 cpu_manager.go:414] "containerMap: removing container" podUID="21c3ed7f-16bd-4693-90f4-93bb8b549bcd" containerName="payload-pre-processing" Oct 31 13:01:47.603537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.597719 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c3ed7f-16bd-4693-90f4-93bb8b549bcd" containerName="payload-pre-processing" Oct 31 13:01:47.603537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.597897 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="83eb243b-c95a-4851-943c-d61e2934157f" containerName="payload-processing" Oct 31 13:01:47.603537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.597912 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c3ed7f-16bd-4693-90f4-93bb8b549bcd" containerName="payload-pre-processing" Oct 31 13:01:47.625073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.624998 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.631717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.626136 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7"] Oct 31 13:01:47.631717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.630967 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4989f9-openshift-default-dockercfg-8md7r\"" Oct 31 13:01:47.694207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694163 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-podinfo\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694410 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-data\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694461 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istiod-ca-cert\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694538 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-envoy\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694611 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694657 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-certs\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694688 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-token\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694777 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-credential-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.694919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.694843 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wpb\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-kube-api-access-m7wpb\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.796956 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-data\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797047 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istiod-ca-cert\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797078 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-envoy\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797147 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797173 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-certs\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797210 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-token\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797248 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-credential-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797319 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wpb\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-kube-api-access-m7wpb\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.797723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.797392 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-podinfo\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.798434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.798254 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-data\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.805668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.801773 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istiod-ca-cert\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.805668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.803225 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.821718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.817787 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-certs\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.821718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.818186 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-credential-socket\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.826796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.826761 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-envoy\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.837672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.827289 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-podinfo\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.976982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.976648 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-token\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:47.980654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:47.978409 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wpb\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-kube-api-access-m7wpb\") pod \"e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:48.062979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.060570 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c3ed7f-16bd-4693-90f4-93bb8b549bcd" path="/var/lib/kubelet/pods/21c3ed7f-16bd-4693-90f4-93bb8b549bcd/volumes" Oct 31 13:01:48.062979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.061318 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec324f3-c445-4548-a102-8ea170d3d269" path="/var/lib/kubelet/pods/6ec324f3-c445-4548-a102-8ea170d3d269/volumes" Oct 31 13:01:48.062979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.061883 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83eb243b-c95a-4851-943c-d61e2934157f" path="/var/lib/kubelet/pods/83eb243b-c95a-4851-943c-d61e2934157f/volumes" Oct 31 13:01:48.063368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.063348 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e" path="/var/lib/kubelet/pods/a3bcf95d-8be5-4343-a9c2-3f2ab5ec519e/volumes" Oct 31 13:01:48.192338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.192293 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd"] Oct 31 13:01:48.198145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.198120 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:48.209837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.209813 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db792\"" Oct 31 13:01:48.221936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.221906 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd"] Oct 31 13:01:48.240921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.240589 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:48.378706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.363131 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad746ba7-5a6e-490c-a683-13fae3671dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:48.380412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.380107 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjcq\" (UniqueName: \"kubernetes.io/projected/ad746ba7-5a6e-490c-a683-13fae3671dd7-kube-api-access-vkjcq\") pod \"payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:48.536708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.532470 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad746ba7-5a6e-490c-a683-13fae3671dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:48.536708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.532533 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjcq\" (UniqueName: \"kubernetes.io/projected/ad746ba7-5a6e-490c-a683-13fae3671dd7-kube-api-access-vkjcq\") pod \"payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:48.536708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.533919 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad746ba7-5a6e-490c-a683-13fae3671dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:48.583873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.581492 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjcq\" (UniqueName: \"kubernetes.io/projected/ad746ba7-5a6e-490c-a683-13fae3671dd7-kube-api-access-vkjcq\") pod \"payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:48.777955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.777920 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7"] Oct 31 13:01:48.779603 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:01:48.779570 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4a5916_fd55_4eb2_bb66_08f1cdd48371.slice/crio-3338c261d8a381c619050fed361bc9bb3b598cf8d58e8e62d9146ebdb4544d5e WatchSource:0}: Error finding container 3338c261d8a381c619050fed361bc9bb3b598cf8d58e8e62d9146ebdb4544d5e: Status 404 returned error can't find the container with id 3338c261d8a381c619050fed361bc9bb3b598cf8d58e8e62d9146ebdb4544d5e Oct 31 13:01:48.783710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.783663 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:48.783826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.783764 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:48.783826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.783813 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:48.820138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:48.820102 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:49.139484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:49.139432 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" event={"ID":"9f4a5916-fd55-4eb2-bb66-08f1cdd48371","Type":"ContainerStarted","Data":"3338c261d8a381c619050fed361bc9bb3b598cf8d58e8e62d9146ebdb4544d5e"} Oct 31 13:01:49.483461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:49.483431 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd"] Oct 31 13:01:49.484090 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:01:49.484054 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad746ba7_5a6e_490c_a683_13fae3671dd7.slice/crio-d8875310ca8da84f3830d4b1a817ac2e718fd109ec682da84d59942263070c7c WatchSource:0}: Error finding container d8875310ca8da84f3830d4b1a817ac2e718fd109ec682da84d59942263070c7c: Status 404 returned error can't find the container with id d8875310ca8da84f3830d4b1a817ac2e718fd109ec682da84d59942263070c7c Oct 31 13:01:50.149447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.149408 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" event={"ID":"9f4a5916-fd55-4eb2-bb66-08f1cdd48371","Type":"ContainerStarted","Data":"02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc"} Oct 31 13:01:50.151232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.151200 5074 generic.go:356] "Generic (PLEG): container finished" podID="3221e591-a5be-469e-85db-5f756c6de8ec" containerID="dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2" exitCode=0 Oct 31 13:01:50.151379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.151270 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" event={"ID":"3221e591-a5be-469e-85db-5f756c6de8ec","Type":"ContainerDied","Data":"dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2"} Oct 31 13:01:50.151738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.151719 5074 scope.go:122] "RemoveContainer" containerID="dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2" Oct 31 13:01:50.153106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.153076 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" event={"ID":"ad746ba7-5a6e-490c-a683-13fae3671dd7","Type":"ContainerStarted","Data":"50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc"} Oct 31 13:01:50.153243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.153131 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" event={"ID":"ad746ba7-5a6e-490c-a683-13fae3671dd7","Type":"ContainerStarted","Data":"d8875310ca8da84f3830d4b1a817ac2e718fd109ec682da84d59942263070c7c"} Oct 31 13:01:50.153243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.153165 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:01:50.153694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.153657 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:50.153803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.153747 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:50.153803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.153794 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:01:50.221332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.220418 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podStartSLOduration=3.220372375 podStartE2EDuration="3.220372375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:50.188797842 +0000 UTC m=+1804.366515859" watchObservedRunningTime="2026-10-31 13:01:50.220372375 +0000 UTC m=+1804.398090362" Oct 31 13:01:50.240276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.240218 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" podStartSLOduration=2.240201688 podStartE2EDuration="2.240201688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:50.235050096 +0000 UTC m=+1804.412768081" watchObservedRunningTime="2026-10-31 13:01:50.240201688 +0000 UTC m=+1804.417919703" Oct 31 13:01:50.243367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.241271 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:50.248389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:50.248083 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:51.350175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:51.350137 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" event={"ID":"3221e591-a5be-469e-85db-5f756c6de8ec","Type":"ContainerStarted","Data":"61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09"} Oct 31 13:01:51.352782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:51.352756 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:51.356383 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:51.356354 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:01:51.453303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:51.453271 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:01:51.454136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:51.454108 5074 patch_prober.go:29] interesting pod/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:01:51.454276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:51.454157 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:01:52.453591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:52.453554 5074 patch_prober.go:29] interesting pod/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:01:52.454143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:52.453616 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:01:53.453262 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:53.453222 5074 patch_prober.go:29] interesting pod/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:01:53.453452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:53.453283 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:01:54.116760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.116711 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn"] Oct 31 13:01:54.128481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.128449 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn"] Oct 31 13:01:54.130506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.128808 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:54.140315 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.140052 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4989f9-dockercfg-gg756\"" Oct 31 13:01:54.174193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.174130 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4989f9\"" Oct 31 13:01:54.415715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.403870 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:54.415715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.403955 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbjt\" (UniqueName: \"kubernetes.io/projected/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-kube-api-access-bbbjt\") pod \"payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:54.459401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.459366 5074 patch_prober.go:29] interesting pod/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:01:54.459701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.459671 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:01:54.505683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.505622 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:54.506029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.506003 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbjt\" (UniqueName: \"kubernetes.io/projected/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-kube-api-access-bbbjt\") pod \"payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:54.506878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.506852 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:54.520303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.520231 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbjt\" (UniqueName: \"kubernetes.io/projected/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-kube-api-access-bbbjt\") pod \"payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:54.753717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:54.753660 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:55.132946 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:01:55.132911 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705dd632_9cca_4faf_ad8b_c6e9bc15ed76.slice/crio-662f23e01109c299532e90e69dee7ed5610062308418102276a86e88ef71212b WatchSource:0}: Error finding container 662f23e01109c299532e90e69dee7ed5610062308418102276a86e88ef71212b: Status 404 returned error can't find the container with id 662f23e01109c299532e90e69dee7ed5610062308418102276a86e88ef71212b Oct 31 13:01:55.134756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:55.134730 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn"] Oct 31 13:01:55.453992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:55.453895 5074 patch_prober.go:29] interesting pod/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:01:55.453992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:55.453957 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:01:55.505309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:55.505263 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" event={"ID":"705dd632-9cca-4faf-ad8b-c6e9bc15ed76","Type":"ContainerStarted","Data":"bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae"} Oct 31 13:01:55.505522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:55.505337 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" event={"ID":"705dd632-9cca-4faf-ad8b-c6e9bc15ed76","Type":"ContainerStarted","Data":"662f23e01109c299532e90e69dee7ed5610062308418102276a86e88ef71212b"} Oct 31 13:01:55.505723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:55.505699 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:01:55.539983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:55.537098 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" podStartSLOduration=1.537075403 podStartE2EDuration="1.537075403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:55.526897094 +0000 UTC m=+1809.704615086" watchObservedRunningTime="2026-10-31 13:01:55.537075403 +0000 UTC m=+1809.714793387" Oct 31 13:01:56.015369 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:01:56.015328 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:01:56.453144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:56.453053 5074 patch_prober.go:29] interesting pod/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:01:56.453144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:56.453107 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:01:57.458227 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:57.458190 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:01:57.458666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:57.458612 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:01:57.459435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:57.459411 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:01:59.041802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:59.041769 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7"] Oct 31 13:01:59.046768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:59.046741 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2af416-z77v7"] Oct 31 13:01:59.941669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:01:59.941593 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f334b4-7f23-4f02-99e6-677095e15c62" path="/var/lib/kubelet/pods/51f334b4-7f23-4f02-99e6-677095e15c62/volumes" Oct 31 13:02:01.353057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:01.353030 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:02:06.480429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:06.480386 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5"] Oct 31 13:02:06.531119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:06.531085 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:02:06.537870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:06.537834 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47"] Oct 31 13:02:11.108425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.108387 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c"] Oct 31 13:02:11.117259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.117218 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af331fab-59x6c"] Oct 31 13:02:11.480910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.480852 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" podUID="acbed071-cd2b-4ca4-9224-2651ce86449f" containerName="payload-processing" containerID="cri-o://1f88fd9a09121fb233500e83a0d6125e974598d76a6bf34659646c74d56ce3ae" gracePeriod=25 Oct 31 13:02:11.539354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.539319 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" podUID="1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" containerName="payload-pre-processing" containerID="cri-o://84a69615f760cfeecdc8e08be508de075808f4959b19e424d9701e52098d78f9" gracePeriod=25 Oct 31 13:02:11.677627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.677054 5074 generic.go:356] "Generic (PLEG): container finished" podID="acbed071-cd2b-4ca4-9224-2651ce86449f" containerID="1f88fd9a09121fb233500e83a0d6125e974598d76a6bf34659646c74d56ce3ae" exitCode=0 Oct 31 13:02:11.677627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.677205 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" event={"ID":"acbed071-cd2b-4ca4-9224-2651ce86449f","Type":"ContainerDied","Data":"1f88fd9a09121fb233500e83a0d6125e974598d76a6bf34659646c74d56ce3ae"} Oct 31 13:02:11.679429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.679378 5074 generic.go:356] "Generic (PLEG): container finished" podID="1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" containerID="84a69615f760cfeecdc8e08be508de075808f4959b19e424d9701e52098d78f9" exitCode=0 Oct 31 13:02:11.679574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.679527 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" event={"ID":"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c","Type":"ContainerDied","Data":"84a69615f760cfeecdc8e08be508de075808f4959b19e424d9701e52098d78f9"} Oct 31 13:02:11.861994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.861962 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:02:11.876824 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.876795 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:02:11.928269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.928237 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbed071-cd2b-4ca4-9224-2651ce86449f-kube-api-access-q85c6\" (UniqueName: \"kubernetes.io/projected/acbed071-cd2b-4ca4-9224-2651ce86449f-kube-api-access-q85c6\") pod \"acbed071-cd2b-4ca4-9224-2651ce86449f\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " Oct 31 13:02:11.928450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.928291 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbed071-cd2b-4ca4-9224-2651ce86449f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbed071-cd2b-4ca4-9224-2651ce86449f-plugins-config-volume\") pod \"acbed071-cd2b-4ca4-9224-2651ce86449f\" (UID: \"acbed071-cd2b-4ca4-9224-2651ce86449f\") " Oct 31 13:02:11.928450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.928326 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-plugins-config-volume\") pod \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " Oct 31 13:02:11.928589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.928537 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-kube-api-access-294v6\" (UniqueName: \"kubernetes.io/projected/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-kube-api-access-294v6\") pod \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\" (UID: \"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c\") " Oct 31 13:02:11.928947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.928915 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-plugins-config-volume" pod "1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" (UID: "1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:11.929381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.929357 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:11.929789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.929765 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbed071-cd2b-4ca4-9224-2651ce86449f-plugins-config-volume" pod "acbed071-cd2b-4ca4-9224-2651ce86449f" (UID: "acbed071-cd2b-4ca4-9224-2651ce86449f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:11.931038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.931010 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbed071-cd2b-4ca4-9224-2651ce86449f-kube-api-access-q85c6" pod "acbed071-cd2b-4ca4-9224-2651ce86449f" (UID: "acbed071-cd2b-4ca4-9224-2651ce86449f"). InnerVolumeSpecName "kube-api-access-q85c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:11.931798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.931749 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-kube-api-access-294v6" pod "1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" (UID: "1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c"). InnerVolumeSpecName "kube-api-access-294v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:11.942249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:11.942221 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474314d9-4d48-49b2-96fc-7cf8e1765a47" path="/var/lib/kubelet/pods/474314d9-4d48-49b2-96fc-7cf8e1765a47/volumes" Oct 31 13:02:12.030510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.030429 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q85c6\" (UniqueName: \"kubernetes.io/projected/acbed071-cd2b-4ca4-9224-2651ce86449f-kube-api-access-q85c6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:12.030510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.030466 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbed071-cd2b-4ca4-9224-2651ce86449f-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:12.030510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.030483 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-294v6\" (UniqueName: \"kubernetes.io/projected/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c-kube-api-access-294v6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:12.696393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.696337 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" Oct 31 13:02:12.696393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.696368 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5" event={"ID":"acbed071-cd2b-4ca4-9224-2651ce86449f","Type":"ContainerDied","Data":"83711368a00e2923fad70fabd4b225859dea717ad9417eb5c304bb6d38c7ecc1"} Oct 31 13:02:12.696970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.696441 5074 scope.go:122] "RemoveContainer" containerID="1f88fd9a09121fb233500e83a0d6125e974598d76a6bf34659646c74d56ce3ae" Oct 31 13:02:12.699201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.699176 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" event={"ID":"1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c","Type":"ContainerDied","Data":"888b9f371ea2c18d896039b7324406228ec836704572c6ae65a198d50f333f4b"} Oct 31 13:02:12.699381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.699271 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47" Oct 31 13:02:12.710124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.710106 5074 scope.go:122] "RemoveContainer" containerID="84a69615f760cfeecdc8e08be508de075808f4959b19e424d9701e52098d78f9" Oct 31 13:02:12.722021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.722001 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5"] Oct 31 13:02:12.725754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.725730 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc2db4-6bdbfc785-92gd5"] Oct 31 13:02:12.737201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.737171 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47"] Oct 31 13:02:12.740556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:12.740537 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc2db4-7b46476bc9-pld47"] Oct 31 13:02:13.941246 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:13.941208 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" path="/var/lib/kubelet/pods/1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c/volumes" Oct 31 13:02:13.941740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:13.941725 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbed071-cd2b-4ca4-9224-2651ce86449f" path="/var/lib/kubelet/pods/acbed071-cd2b-4ca4-9224-2651ce86449f/volumes" Oct 31 13:02:16.857945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:16.857910 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd"] Oct 31 13:02:17.669649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:17.669610 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:02:20.929536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:20.929497 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj"] Oct 31 13:02:20.943479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:20.943449 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj"] Oct 31 13:02:21.872681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:21.860170 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" podUID="ad746ba7-5a6e-490c-a683-13fae3671dd7" containerName="payload-pre-processing" containerID="cri-o://50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc" gracePeriod=25 Oct 31 13:02:22.316564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.316020 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:02:22.500708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.497336 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad746ba7-5a6e-490c-a683-13fae3671dd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad746ba7-5a6e-490c-a683-13fae3671dd7-plugins-config-volume\") pod \"ad746ba7-5a6e-490c-a683-13fae3671dd7\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " Oct 31 13:02:22.500708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.497405 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad746ba7-5a6e-490c-a683-13fae3671dd7-kube-api-access-vkjcq\" (UniqueName: \"kubernetes.io/projected/ad746ba7-5a6e-490c-a683-13fae3671dd7-kube-api-access-vkjcq\") pod \"ad746ba7-5a6e-490c-a683-13fae3671dd7\" (UID: \"ad746ba7-5a6e-490c-a683-13fae3671dd7\") " Oct 31 13:02:22.500708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.498939 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad746ba7-5a6e-490c-a683-13fae3671dd7-plugins-config-volume" pod "ad746ba7-5a6e-490c-a683-13fae3671dd7" (UID: "ad746ba7-5a6e-490c-a683-13fae3671dd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:22.502188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.502150 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad746ba7-5a6e-490c-a683-13fae3671dd7-kube-api-access-vkjcq" pod "ad746ba7-5a6e-490c-a683-13fae3671dd7" (UID: "ad746ba7-5a6e-490c-a683-13fae3671dd7"). InnerVolumeSpecName "kube-api-access-vkjcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:22.605015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.604928 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad746ba7-5a6e-490c-a683-13fae3671dd7-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.605015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.604966 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjcq\" (UniqueName: \"kubernetes.io/projected/ad746ba7-5a6e-490c-a683-13fae3671dd7-kube-api-access-vkjcq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:22.936956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.936020 5074 generic.go:356] "Generic (PLEG): container finished" podID="ad746ba7-5a6e-490c-a683-13fae3671dd7" containerID="50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc" exitCode=0 Oct 31 13:02:22.936956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.936346 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" event={"ID":"ad746ba7-5a6e-490c-a683-13fae3671dd7","Type":"ContainerDied","Data":"50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc"} Oct 31 13:02:22.936956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.936395 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" Oct 31 13:02:22.936956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.936917 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd" event={"ID":"ad746ba7-5a6e-490c-a683-13fae3671dd7","Type":"ContainerDied","Data":"d8875310ca8da84f3830d4b1a817ac2e718fd109ec682da84d59942263070c7c"} Oct 31 13:02:22.936956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.936957 5074 scope.go:122] "RemoveContainer" containerID="50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc" Oct 31 13:02:22.959453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.959423 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn"] Oct 31 13:02:22.961314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.961300 5074 scope.go:122] "RemoveContainer" containerID="50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc" Oct 31 13:02:22.961840 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:02:22.961818 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc\": container with ID starting with 50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc not found: ID does not exist" containerID="50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc" Oct 31 13:02:22.961929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:22.961844 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc"} err="failed to get container status \"50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc\": rpc error: code = NotFound desc = could not find container \"50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc\": container with ID starting with 50fd618c7c881eb5e4a084ddfe037b4b143cd0bc79b313f4a31ed2e9eedbf0cc not found: ID does not exist" Oct 31 13:02:23.042172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:23.042136 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd"] Oct 31 13:02:23.047118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:23.047089 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db792-f787676bc-hqtxd"] Oct 31 13:02:23.945774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:23.945738 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad746ba7-5a6e-490c-a683-13fae3671dd7" path="/var/lib/kubelet/pods/ad746ba7-5a6e-490c-a683-13fae3671dd7/volumes" Oct 31 13:02:25.979661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:25.958654 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" podUID="9ed8daef-e8f4-4fac-a139-a9a557b365a5" containerName="payload-pre-processing" containerID="cri-o://e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851" gracePeriod=25 Oct 31 13:02:25.979661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:25.958828 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" podUID="7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" containerName="payload-processing" containerID="cri-o://e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a" gracePeriod=25 Oct 31 13:02:26.071605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.071567 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn"] Oct 31 13:02:26.071881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.071865 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f22068-qtnfn"] Oct 31 13:02:26.151304 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:02:26.151231 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:02:26.436678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.433794 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:02:26.436899 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.436729 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:02:26.499432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.499336 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed8daef-e8f4-4fac-a139-a9a557b365a5-kube-api-access-9gksx\" (UniqueName: \"kubernetes.io/projected/9ed8daef-e8f4-4fac-a139-a9a557b365a5-kube-api-access-9gksx\") pod \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " Oct 31 13:02:26.499727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.499450 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed8daef-e8f4-4fac-a139-a9a557b365a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8daef-e8f4-4fac-a139-a9a557b365a5-plugins-config-volume\") pod \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\" (UID: \"9ed8daef-e8f4-4fac-a139-a9a557b365a5\") " Oct 31 13:02:26.499727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.499596 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-kube-api-access-8q64d\" (UniqueName: \"kubernetes.io/projected/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-kube-api-access-8q64d\") pod \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " Oct 31 13:02:26.499727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.499624 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-plugins-config-volume\") pod \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\" (UID: \"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8\") " Oct 31 13:02:26.500627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.500594 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-plugins-config-volume" pod "7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" (UID: "7d9ceac4-b56e-4edb-b5a6-36cad784b8f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:26.501725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.501694 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed8daef-e8f4-4fac-a139-a9a557b365a5-plugins-config-volume" pod "9ed8daef-e8f4-4fac-a139-a9a557b365a5" (UID: "9ed8daef-e8f4-4fac-a139-a9a557b365a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:26.523349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.520919 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-kube-api-access-8q64d" pod "7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" (UID: "7d9ceac4-b56e-4edb-b5a6-36cad784b8f8"). InnerVolumeSpecName "kube-api-access-8q64d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:26.523349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.522213 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed8daef-e8f4-4fac-a139-a9a557b365a5-kube-api-access-9gksx" pod "9ed8daef-e8f4-4fac-a139-a9a557b365a5" (UID: "9ed8daef-e8f4-4fac-a139-a9a557b365a5"). InnerVolumeSpecName "kube-api-access-9gksx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:26.605481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.604948 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gksx\" (UniqueName: \"kubernetes.io/projected/9ed8daef-e8f4-4fac-a139-a9a557b365a5-kube-api-access-9gksx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:26.605481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.604979 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8daef-e8f4-4fac-a139-a9a557b365a5-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:26.605481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.604989 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q64d\" (UniqueName: \"kubernetes.io/projected/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-kube-api-access-8q64d\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:26.605481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:26.604997 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:27.040854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.040818 5074 generic.go:356] "Generic (PLEG): container finished" podID="7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" containerID="e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a" exitCode=0 Oct 31 13:02:27.041340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.040870 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" event={"ID":"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8","Type":"ContainerDied","Data":"e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a"} Oct 31 13:02:27.041340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.040880 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" Oct 31 13:02:27.041340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.040959 5074 scope.go:122] "RemoveContainer" containerID="e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a" Oct 31 13:02:27.041340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.040945 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj" event={"ID":"7d9ceac4-b56e-4edb-b5a6-36cad784b8f8","Type":"ContainerDied","Data":"30ae9b47b6ba447ab6f939c4534596f624912ab1f303e7e8d090153e2c2e9f78"} Oct 31 13:02:27.042773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.042750 5074 generic.go:356] "Generic (PLEG): container finished" podID="9ed8daef-e8f4-4fac-a139-a9a557b365a5" containerID="e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851" exitCode=0 Oct 31 13:02:27.042994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.042957 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" event={"ID":"9ed8daef-e8f4-4fac-a139-a9a557b365a5","Type":"ContainerDied","Data":"e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851"} Oct 31 13:02:27.043102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.043004 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" event={"ID":"9ed8daef-e8f4-4fac-a139-a9a557b365a5","Type":"ContainerDied","Data":"d7a94b5c4e2472e4cb6e0044eb492b9edf8d5d7733dd9d8b5e2fcee1549fe06f"} Oct 31 13:02:27.043184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.043102 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj" Oct 31 13:02:27.065208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.065177 5074 scope.go:122] "RemoveContainer" containerID="e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a" Oct 31 13:02:27.065823 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:02:27.065658 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a\": container with ID starting with e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a not found: ID does not exist" containerID="e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a" Oct 31 13:02:27.065823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.065700 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a"} err="failed to get container status \"e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a\": rpc error: code = NotFound desc = could not find container \"e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a\": container with ID starting with e826ba865e0ab84a1096da85f8718208c0ab21e48c56f26e08d2151de16f0a0a not found: ID does not exist" Oct 31 13:02:27.065823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.065725 5074 scope.go:122] "RemoveContainer" containerID="e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851" Oct 31 13:02:27.119973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.119843 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj"] Oct 31 13:02:27.124834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.124802 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-01e8d2-586cd8fd49-d2djj"] Oct 31 13:02:27.133351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.132963 5074 scope.go:122] "RemoveContainer" containerID="e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851" Oct 31 13:02:27.135090 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:02:27.133955 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851\": container with ID starting with e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851 not found: ID does not exist" containerID="e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851" Oct 31 13:02:27.135090 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.134005 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851"} err="failed to get container status \"e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851\": rpc error: code = NotFound desc = could not find container \"e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851\": container with ID starting with e6604bd5d96feeabd0da76bbcdb8975e43ac994b154fe9983a77482f2653e851 not found: ID does not exist" Oct 31 13:02:27.150302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.149715 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj"] Oct 31 13:02:27.155661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.155602 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-01e8d2-7cc9949c9-p45zj"] Oct 31 13:02:27.942887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.942851 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" path="/var/lib/kubelet/pods/7d9ceac4-b56e-4edb-b5a6-36cad784b8f8/volumes" Oct 31 13:02:27.943374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.943356 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed8daef-e8f4-4fac-a139-a9a557b365a5" path="/var/lib/kubelet/pods/9ed8daef-e8f4-4fac-a139-a9a557b365a5/volumes" Oct 31 13:02:27.943810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.943794 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad219e8-9e0e-44fc-ae40-88a218b8f760" path="/var/lib/kubelet/pods/fad219e8-9e0e-44fc-ae40-88a218b8f760/volumes" Oct 31 13:02:27.960863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:27.960824 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" podUID="705dd632-9cca-4faf-ad8b-c6e9bc15ed76" containerName="payload-processing" containerID="cri-o://bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae" gracePeriod=25 Oct 31 13:02:28.321550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:28.321521 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:02:28.443620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:28.443584 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-plugins-config-volume\") pod \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " Oct 31 13:02:28.443845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:28.443829 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-kube-api-access-bbbjt\" (UniqueName: \"kubernetes.io/projected/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-kube-api-access-bbbjt\") pod \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\" (UID: \"705dd632-9cca-4faf-ad8b-c6e9bc15ed76\") " Oct 31 13:02:28.445680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:28.444490 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-plugins-config-volume" pod "705dd632-9cca-4faf-ad8b-c6e9bc15ed76" (UID: "705dd632-9cca-4faf-ad8b-c6e9bc15ed76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:02:28.445680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:28.445417 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:28.458586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:28.458549 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-kube-api-access-bbbjt" pod "705dd632-9cca-4faf-ad8b-c6e9bc15ed76" (UID: "705dd632-9cca-4faf-ad8b-c6e9bc15ed76"). InnerVolumeSpecName "kube-api-access-bbbjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:02:28.547325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:28.547249 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbbjt\" (UniqueName: \"kubernetes.io/projected/705dd632-9cca-4faf-ad8b-c6e9bc15ed76-kube-api-access-bbbjt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:02:29.068814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.068778 5074 generic.go:356] "Generic (PLEG): container finished" podID="705dd632-9cca-4faf-ad8b-c6e9bc15ed76" containerID="bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae" exitCode=0 Oct 31 13:02:29.069017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.068870 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" event={"ID":"705dd632-9cca-4faf-ad8b-c6e9bc15ed76","Type":"ContainerDied","Data":"bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae"} Oct 31 13:02:29.069017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.068906 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" event={"ID":"705dd632-9cca-4faf-ad8b-c6e9bc15ed76","Type":"ContainerDied","Data":"662f23e01109c299532e90e69dee7ed5610062308418102276a86e88ef71212b"} Oct 31 13:02:29.069017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.068926 5074 scope.go:122] "RemoveContainer" containerID="bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae" Oct 31 13:02:29.069194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.069046 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn" Oct 31 13:02:29.087133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.087107 5074 scope.go:122] "RemoveContainer" containerID="bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae" Oct 31 13:02:29.087473 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:02:29.087448 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae\": container with ID starting with bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae not found: ID does not exist" containerID="bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae" Oct 31 13:02:29.087572 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.087480 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae"} err="failed to get container status \"bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae\": rpc error: code = NotFound desc = could not find container \"bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae\": container with ID starting with bf4fa9009efe7ee01270a77acedb0301a2a9504b944b43b0685463096cd2c7ae not found: ID does not exist" Oct 31 13:02:29.100027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.099999 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn"] Oct 31 13:02:29.103500 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.103479 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4989f9-77b8db48d7-q7jcn"] Oct 31 13:02:29.942305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:29.942260 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705dd632-9cca-4faf-ad8b-c6e9bc15ed76" path="/var/lib/kubelet/pods/705dd632-9cca-4faf-ad8b-c6e9bc15ed76/volumes" Oct 31 13:02:31.090808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:31.090775 5074 generic.go:356] "Generic (PLEG): container finished" podID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerID="02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc" exitCode=0 Oct 31 13:02:31.091244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:31.090826 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" event={"ID":"9f4a5916-fd55-4eb2-bb66-08f1cdd48371","Type":"ContainerDied","Data":"02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc"} Oct 31 13:02:31.091288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:31.091274 5074 scope.go:122] "RemoveContainer" containerID="02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc" Oct 31 13:02:31.092972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:31.092941 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:31.093088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:31.093018 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:31.093088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:31.093056 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:02:32.108274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:32.108232 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" event={"ID":"9f4a5916-fd55-4eb2-bb66-08f1cdd48371","Type":"ContainerStarted","Data":"3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc"} Oct 31 13:02:32.241118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:32.241076 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:02:32.243470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:32.243432 5074 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 13:02:32.243691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:32.243491 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 13:02:33.248553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:33.247834 5074 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 13:02:33.248553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:33.247897 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 13:02:34.241832 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:34.241791 5074 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 13:02:34.242056 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:34.241883 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 13:02:35.241470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:35.241439 5074 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 13:02:35.241904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:35.241495 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 13:02:36.249659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:36.247147 5074 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 13:02:36.249659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:36.247212 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 13:02:36.353667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:36.352817 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:02:37.241197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:37.241161 5074 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 13:02:37.241433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:37.241217 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 13:02:38.047232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:38.047194 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x"] Oct 31 13:02:38.052456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:38.052411 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdb78e7-f657x"] Oct 31 13:02:38.246170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:38.246131 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:02:38.247289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:38.247266 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:02:39.946397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:39.946362 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0371edb-ad02-4fe8-b6ac-a921cf0e1720" path="/var/lib/kubelet/pods/d0371edb-ad02-4fe8-b6ac-a921cf0e1720/volumes" Oct 31 13:02:46.591727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.591688 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:02:46.910186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.910097 5074 scope.go:122] "RemoveContainer" containerID="938b8adc1180de771f9cc5a578147905172e2ca935292d47784ab28488c09755" Oct 31 13:02:46.931974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.928824 5074 scope.go:122] "RemoveContainer" containerID="3e2aac30688b6b80a245e37224ab685cc548d0ca7a627241c3c50685f95dcaec" Oct 31 13:02:46.941131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.941099 5074 scope.go:122] "RemoveContainer" containerID="b27b44a3b0d763a7920026f674b9415601e1cd98628d466733151806f9eecd64" Oct 31 13:02:46.953527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.953377 5074 scope.go:122] "RemoveContainer" containerID="51825be5a7a3f9e8f0667d17c20b1559fee8d7a2fc6d3e03bd17401035da82ff" Oct 31 13:02:46.968125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.968102 5074 scope.go:122] "RemoveContainer" containerID="dd6d633c4d0fd72ba268deb67f61ef2b7702d1f3bd9a24bf67968a7636d3e530" Oct 31 13:02:46.978675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.978651 5074 scope.go:122] "RemoveContainer" containerID="e3bf0776ad21ae6e0626a31e6558e2ff00f37274067a5d917fdef531cef00238" Oct 31 13:02:46.987414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.987394 5074 scope.go:122] "RemoveContainer" containerID="c9c75a050e6815f30a87815c295560ef8763fd519f0a41401bf55f6cb1eaee03" Oct 31 13:02:46.996609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:46.996584 5074 scope.go:122] "RemoveContainer" containerID="a0356c07f028bdf0879b7b40afe84a169f71ece5353df0f090bd7373a9e57aef" Oct 31 13:02:47.005549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:47.005530 5074 scope.go:122] "RemoveContainer" containerID="beaf9531a4c2f116e0cd057c9ca4b879638534fe610f9115029b9e2a4b9f72c8" Oct 31 13:02:47.015104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:47.015085 5074 scope.go:122] "RemoveContainer" containerID="ff0a313fd3084db0ddc9c700657eab47a872699981b7536dea6da6a2f796bf87" Oct 31 13:02:47.024037 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:47.024019 5074 scope.go:122] "RemoveContainer" containerID="b74878cfe39219c6822af49e33f6bcdeb7d519ca5e06477aba823d304b292ff7" Oct 31 13:02:47.032882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:47.032863 5074 scope.go:122] "RemoveContainer" containerID="d6232fa45375271469796922e72a259422f1af6fe406f9151c15ddf52a9fef45" Oct 31 13:02:47.041531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:47.041509 5074 scope.go:122] "RemoveContainer" containerID="6c6bdebe64b0c787bef244411e33475a58c4ecfb0b137ed7e36ea81e834a92db" Oct 31 13:02:50.051840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:50.051803 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs"] Oct 31 13:02:50.061569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:50.061536 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264abcf4-wlnbs"] Oct 31 13:02:51.946098 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:51.946047 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029134c2-ae8e-45b9-8ffb-a800c223758f" path="/var/lib/kubelet/pods/029134c2-ae8e-45b9-8ffb-a800c223758f/volumes" Oct 31 13:02:56.001813 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:02:56.001769 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:02:56.331855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:56.331750 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w"] Oct 31 13:02:56.332082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:56.332027 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" containerID="cri-o://61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09" gracePeriod=30 Oct 31 13:02:57.462067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:57.462021 5074 patch_prober.go:29] interesting pod/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:02:57.462462 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:02:57.462088 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:03:00.072922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:00.072887 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj"] Oct 31 13:03:00.081244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:00.081200 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc55c842-h88jj"] Oct 31 13:03:01.702780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.702746 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:03:01.805577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805541 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-certs\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.805809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805595 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-kube-api-access-s2826\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-kube-api-access-s2826\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.805809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805707 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-socket\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.805809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805794 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-istio-token\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-istio-token\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.805988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805845 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-data\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.805988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805880 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-certs" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:01.805988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805933 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-credential-socket\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.805988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.805973 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-envoy\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.806188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.806041 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3221e591-a5be-469e-85db-5f756c6de8ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3221e591-a5be-469e-85db-5f756c6de8ec-istiod-ca-cert\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.806188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.806105 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3221e591-a5be-469e-85db-5f756c6de8ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3221e591-a5be-469e-85db-5f756c6de8ec-istio-podinfo\") pod \"3221e591-a5be-469e-85db-5f756c6de8ec\" (UID: \"3221e591-a5be-469e-85db-5f756c6de8ec\") " Oct 31 13:03:01.806296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.806251 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-credential-socket" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:01.806683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.806367 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-data" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:01.806823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.806778 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-socket" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:01.806891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.806814 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3221e591-a5be-469e-85db-5f756c6de8ec-istiod-ca-cert" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:01.807009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.806985 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.807174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.807156 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.807271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.807259 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.807362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.807349 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.807447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.807435 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3221e591-a5be-469e-85db-5f756c6de8ec-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.808432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.808406 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-istio-token" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:01.808786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.808766 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3221e591-a5be-469e-85db-5f756c6de8ec-istio-podinfo" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:03:01.809047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.809025 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-kube-api-access-s2826" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "kube-api-access-s2826". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:01.809147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.809073 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-envoy" pod "3221e591-a5be-469e-85db-5f756c6de8ec" (UID: "3221e591-a5be-469e-85db-5f756c6de8ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:01.899903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.899863 5074 generic.go:356] "Generic (PLEG): container finished" podID="3221e591-a5be-469e-85db-5f756c6de8ec" containerID="61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09" exitCode=0 Oct 31 13:03:01.900101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.900072 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" event={"ID":"3221e591-a5be-469e-85db-5f756c6de8ec","Type":"ContainerDied","Data":"61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09"} Oct 31 13:03:01.900168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.900158 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" event={"ID":"3221e591-a5be-469e-85db-5f756c6de8ec","Type":"ContainerDied","Data":"74e3979c1d6af25454d36ee7d202dd941ca26e6f3a3a62d2dfbc711d3d665f55"} Oct 31 13:03:01.900230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.900189 5074 scope.go:122] "RemoveContainer" containerID="61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09" Oct 31 13:03:01.900417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.900402 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w" Oct 31 13:03:01.910822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.910755 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.910991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.910968 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3221e591-a5be-469e-85db-5f756c6de8ec-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.910991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.910992 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3221e591-a5be-469e-85db-5f756c6de8ec-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.911128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.911003 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2826\" (UniqueName: \"kubernetes.io/projected/3221e591-a5be-469e-85db-5f756c6de8ec-kube-api-access-s2826\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:01.919762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.919740 5074 scope.go:122] "RemoveContainer" containerID="dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2" Oct 31 13:03:01.932238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.932215 5074 scope.go:122] "RemoveContainer" containerID="61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09" Oct 31 13:03:01.932763 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:03:01.932735 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09\": container with ID starting with 61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09 not found: ID does not exist" containerID="61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09" Oct 31 13:03:01.932865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.932774 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09"} err="failed to get container status \"61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09\": rpc error: code = NotFound desc = could not find container \"61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09\": container with ID starting with 61e50d0b41c030fe9f992b4e03cc77c679711c127bff3863b371e2c068e76e09 not found: ID does not exist" Oct 31 13:03:01.932865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.932797 5074 scope.go:122] "RemoveContainer" containerID="dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2" Oct 31 13:03:01.933118 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:03:01.933095 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2\": container with ID starting with dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2 not found: ID does not exist" containerID="dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2" Oct 31 13:03:01.933196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.933128 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2"} err="failed to get container status \"dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2\": rpc error: code = NotFound desc = could not find container \"dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2\": container with ID starting with dae46feb5b6532330eca6e42807d304996c70a189b297a5302ff097ad52254b2 not found: ID does not exist" Oct 31 13:03:01.939838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.939811 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5" path="/var/lib/kubelet/pods/c1a975f5-9ac2-43c6-a3a7-49b3dd66d1f5/volumes" Oct 31 13:03:01.941788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.941762 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w"] Oct 31 13:03:01.944731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:01.944710 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-48dcc1-openshift-default-76c4bf48f5-8m92w"] Oct 31 13:03:03.939114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:03.939081 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" path="/var/lib/kubelet/pods/3221e591-a5be-469e-85db-5f756c6de8ec/volumes" Oct 31 13:03:12.040715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:12.040666 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw"] Oct 31 13:03:12.043866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:12.043841 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe4c330e-n69rw"] Oct 31 13:03:13.939905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:13.939871 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad001eb-4058-4495-abf0-20aefbc9dd21" path="/var/lib/kubelet/pods/aad001eb-4058-4495-abf0-20aefbc9dd21/volumes" Oct 31 13:03:23.047529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:23.047494 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs"] Oct 31 13:03:23.059666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:23.059621 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dff3e8-xmfrs"] Oct 31 13:03:24.037835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:24.037786 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ff4cad-7f28-4999-8fdd-c3c63ab58870" path="/var/lib/kubelet/pods/f9ff4cad-7f28-4999-8fdd-c3c63ab58870/volumes" Oct 31 13:03:25.985750 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:03:25.985706 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:03:30.487936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:30.487881 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:03:30.789074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:30.788993 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv"] Oct 31 13:03:30.789281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:30.789253 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" containerID="cri-o://c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a" gracePeriod=30 Oct 31 13:03:36.186460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.186126 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 13:03:36.253765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.253712 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-certs\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.253930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.253824 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-kube-api-access-625zw\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-kube-api-access-625zw\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.253930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.253870 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-token\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-token\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.254064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.254004 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-socket\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.254129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.254056 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-certs" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:36.254340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.254314 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-socket" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:36.254534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.254339 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-data\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.254777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.254759 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac29fcc-c69a-45b2-8189-70d35d8f4765-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac29fcc-c69a-45b2-8189-70d35d8f4765-istiod-ca-cert\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.255372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.255352 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-credential-socket\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.255825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.255799 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-envoy\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.255934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.254552 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-data" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:36.255934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.255849 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-podinfo\") pod \"cac29fcc-c69a-45b2-8189-70d35d8f4765\" (UID: \"cac29fcc-c69a-45b2-8189-70d35d8f4765\") " Oct 31 13:03:36.256458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.256430 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.256569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.256459 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.256569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.256475 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.256569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.255229 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac29fcc-c69a-45b2-8189-70d35d8f4765-istiod-ca-cert" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:03:36.256569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.255737 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-credential-socket" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:36.256569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.256442 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-kube-api-access-625zw" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "kube-api-access-625zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:36.256888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.256732 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-token" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:03:36.258534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.258479 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-podinfo" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:03:36.260059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.260037 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-envoy" pod "cac29fcc-c69a-45b2-8189-70d35d8f4765" (UID: "cac29fcc-c69a-45b2-8189-70d35d8f4765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:03:36.283948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.282295 5074 generic.go:356] "Generic (PLEG): container finished" podID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerID="c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a" exitCode=0 Oct 31 13:03:36.283948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.282348 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" event={"ID":"cac29fcc-c69a-45b2-8189-70d35d8f4765","Type":"ContainerDied","Data":"c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a"} Oct 31 13:03:36.283948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.282395 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" event={"ID":"cac29fcc-c69a-45b2-8189-70d35d8f4765","Type":"ContainerDied","Data":"7aaf2763a286894dd4544b8502cc64d40c1ca0a57830e68cc39c356412b16211"} Oct 31 13:03:36.283948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.282423 5074 scope.go:122] "RemoveContainer" containerID="c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a" Oct 31 13:03:36.283948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.282586 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv" Oct 31 13:03:36.322334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.322305 5074 scope.go:122] "RemoveContainer" containerID="c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a" Oct 31 13:03:36.322753 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:03:36.322724 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a\": container with ID starting with c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a not found: ID does not exist" containerID="c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a" Oct 31 13:03:36.322841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.322775 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a"} err="failed to get container status \"c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a\": rpc error: code = NotFound desc = could not find container \"c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a\": container with ID starting with c3187085087acf71aa12c1a0f635d27c1a891b47e1f4407aba7c169b0756843a not found: ID does not exist" Oct 31 13:03:36.362685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.358190 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac29fcc-c69a-45b2-8189-70d35d8f4765-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.362685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.358222 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.362685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.358240 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.362685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.358264 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.362685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.358280 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625zw\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-kube-api-access-625zw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.362685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.358294 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac29fcc-c69a-45b2-8189-70d35d8f4765-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:03:36.392137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.392099 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv"] Oct 31 13:03:36.400033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.399946 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-hfrhv"] Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.819372 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn"] Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820277 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820299 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820329 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820337 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820348 5074 cpu_manager.go:414] "containerMap: removing container" podUID="705dd632-9cca-4faf-ad8b-c6e9bc15ed76" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820357 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705dd632-9cca-4faf-ad8b-c6e9bc15ed76" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820393 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820400 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820419 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820426 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820459 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ad746ba7-5a6e-490c-a683-13fae3671dd7" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820467 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad746ba7-5a6e-490c-a683-13fae3671dd7" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820486 5074 cpu_manager.go:414] "containerMap: removing container" podUID="acbed071-cd2b-4ca4-9224-2651ce86449f" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820495 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbed071-cd2b-4ca4-9224-2651ce86449f" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820505 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820512 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820524 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9ed8daef-e8f4-4fac-a139-a9a557b365a5" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820533 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed8daef-e8f4-4fac-a139-a9a557b365a5" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820692 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9ceac4-b56e-4edb-b5a6-36cad784b8f8" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820710 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820724 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad746ba7-5a6e-490c-a683-13fae3671dd7" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820736 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbed071-cd2b-4ca4-9224-2651ce86449f" containerName="payload-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820745 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f02e5f8-2035-4a2c-8b0f-6cee78c4b34c" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820755 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed8daef-e8f4-4fac-a139-a9a557b365a5" containerName="payload-pre-processing" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820766 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3221e591-a5be-469e-85db-5f756c6de8ec" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820777 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" containerName="istio-proxy" Oct 31 13:03:36.823359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.820785 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="705dd632-9cca-4faf-ad8b-c6e9bc15ed76" containerName="payload-processing" Oct 31 13:03:36.826566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.826545 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:36.846030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.846005 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn"] Oct 31 13:03:36.919013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.917111 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c347ff-dockercfg-7q7ht\"" Oct 31 13:03:36.919013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.917216 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c347ff\"" Oct 31 13:03:36.982026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.981988 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cx4c\" (UniqueName: \"kubernetes.io/projected/a97874aa-95e6-4051-8117-e8f0ccaeeb42-kube-api-access-6cx4c\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:36.988843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:36.988798 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97874aa-95e6-4051-8117-e8f0ccaeeb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:37.091050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:37.090951 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cx4c\" (UniqueName: \"kubernetes.io/projected/a97874aa-95e6-4051-8117-e8f0ccaeeb42-kube-api-access-6cx4c\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:37.100981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:37.100905 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97874aa-95e6-4051-8117-e8f0ccaeeb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:37.102823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:37.102758 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97874aa-95e6-4051-8117-e8f0ccaeeb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:37.113324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:37.106985 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cx4c\" (UniqueName: \"kubernetes.io/projected/a97874aa-95e6-4051-8117-e8f0ccaeeb42-kube-api-access-6cx4c\") pod \"payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:37.161470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:37.160224 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:37.476587 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:03:37.476557 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97874aa_95e6_4051_8117_e8f0ccaeeb42.slice/crio-293619ed019b2afdfcb9133f4a051e842c946ba1306c57e5d2a93d48156f223e WatchSource:0}: Error finding container 293619ed019b2afdfcb9133f4a051e842c946ba1306c57e5d2a93d48156f223e: Status 404 returned error can't find the container with id 293619ed019b2afdfcb9133f4a051e842c946ba1306c57e5d2a93d48156f223e Oct 31 13:03:37.476941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:37.476901 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn"] Oct 31 13:03:37.940836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:37.940738 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac29fcc-c69a-45b2-8189-70d35d8f4765" path="/var/lib/kubelet/pods/cac29fcc-c69a-45b2-8189-70d35d8f4765/volumes" Oct 31 13:03:38.306319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:38.306279 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" event={"ID":"a97874aa-95e6-4051-8117-e8f0ccaeeb42","Type":"ContainerStarted","Data":"12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f"} Oct 31 13:03:38.306517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:38.306340 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" event={"ID":"a97874aa-95e6-4051-8117-e8f0ccaeeb42","Type":"ContainerStarted","Data":"293619ed019b2afdfcb9133f4a051e842c946ba1306c57e5d2a93d48156f223e"} Oct 31 13:03:38.306517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:38.306365 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:38.336289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:38.336240 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" podStartSLOduration=2.336227107 podStartE2EDuration="2.336227107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:38.334075171 +0000 UTC m=+1912.511793155" watchObservedRunningTime="2026-10-31 13:03:38.336227107 +0000 UTC m=+1912.513945088" Oct 31 13:03:43.243511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.243469 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf"] Oct 31 13:03:43.248201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.248177 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.265027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.265006 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-49bffb-openshift-default-dockercfg-7qsc9\"" Oct 31 13:03:43.282759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.282728 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf"] Oct 31 13:03:43.321744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.321699 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322034 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322134 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322176 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322218 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcr6\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-kube-api-access-4fcr6\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322278 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322308 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322329 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.322404 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.322377 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423269 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423322 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423345 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423383 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423435 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423487 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423532 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423566 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423618 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcr6\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-kube-api-access-4fcr6\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423790 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423850 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-certs\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.423960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.423927 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-credential-socket\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.424118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.424095 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-data\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.424381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.424363 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istiod-ca-cert\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.425603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.425577 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-envoy\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.426018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.425991 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-podinfo\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.439359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.439331 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcr6\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-kube-api-access-4fcr6\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.442129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.442112 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-token\") pod \"e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.563186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.563099 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:43.880133 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:03:43.880104 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfba18a_aab1_4e7b_ba83_55ead3e05acc.slice/crio-27e1f20334e2d057b929d989ecdb0d29835438b4e7c0818c37b566fa159c2e37 WatchSource:0}: Error finding container 27e1f20334e2d057b929d989ecdb0d29835438b4e7c0818c37b566fa159c2e37: Status 404 returned error can't find the container with id 27e1f20334e2d057b929d989ecdb0d29835438b4e7c0818c37b566fa159c2e37 Oct 31 13:03:43.880242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.880200 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf"] Oct 31 13:03:43.883258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.883225 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:43.883347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.883295 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:43.883347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:43.883326 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:03:44.293385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.293342 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4"] Oct 31 13:03:44.297801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.297776 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.307888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.307860 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fa2079\"" Oct 31 13:03:44.317736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.317700 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4"] Oct 31 13:03:44.337084 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.337048 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdm7\" (UniqueName: \"kubernetes.io/projected/277334af-3358-445e-96c5-8c6cb9b92023-kube-api-access-9jdm7\") pod \"payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.337248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.337118 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277334af-3358-445e-96c5-8c6cb9b92023-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.340290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.340266 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw"] Oct 31 13:03:44.344603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.344584 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:44.362718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.362694 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fa2079-dockercfg-vkf5q\"" Oct 31 13:03:44.367696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.367666 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw"] Oct 31 13:03:44.369075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.369047 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" event={"ID":"dbfba18a-aab1-4e7b-ba83-55ead3e05acc","Type":"ContainerStarted","Data":"3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea"} Oct 31 13:03:44.369176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.369102 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" event={"ID":"dbfba18a-aab1-4e7b-ba83-55ead3e05acc","Type":"ContainerStarted","Data":"27e1f20334e2d057b929d989ecdb0d29835438b4e7c0818c37b566fa159c2e37"} Oct 31 13:03:44.424494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.424402 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podStartSLOduration=1.424379949 podStartE2EDuration="1.424379949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:44.420604327 +0000 UTC m=+1918.598322315" watchObservedRunningTime="2026-10-31 13:03:44.424379949 +0000 UTC m=+1918.602097934" Oct 31 13:03:44.438804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.438765 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ead7707-8656-4809-82d6-d9f2f24581e8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:44.439282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.439258 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm7kg\" (UniqueName: \"kubernetes.io/projected/2ead7707-8656-4809-82d6-d9f2f24581e8-kube-api-access-vm7kg\") pod \"payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:44.439571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.439553 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdm7\" (UniqueName: \"kubernetes.io/projected/277334af-3358-445e-96c5-8c6cb9b92023-kube-api-access-9jdm7\") pod \"payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.441137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.441115 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277334af-3358-445e-96c5-8c6cb9b92023-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.442457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.442097 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277334af-3358-445e-96c5-8c6cb9b92023-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.547704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.545650 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ead7707-8656-4809-82d6-d9f2f24581e8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:44.547704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.545897 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm7kg\" (UniqueName: \"kubernetes.io/projected/2ead7707-8656-4809-82d6-d9f2f24581e8-kube-api-access-vm7kg\") pod \"payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:44.547704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.547336 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ead7707-8656-4809-82d6-d9f2f24581e8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:44.580082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.580051 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:44.582730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.582700 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdm7\" (UniqueName: \"kubernetes.io/projected/277334af-3358-445e-96c5-8c6cb9b92023-kube-api-access-9jdm7\") pod \"payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.583012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.582990 5074 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:03:44.584142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.584114 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 13:03:44.596656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.593490 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm7kg\" (UniqueName: \"kubernetes.io/projected/2ead7707-8656-4809-82d6-d9f2f24581e8-kube-api-access-vm7kg\") pod \"payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:44.625654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.622232 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:44.668988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:44.668946 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:45.043745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.043719 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4"] Oct 31 13:03:45.044985 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:03:45.044956 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277334af_3358_445e_96c5_8c6cb9b92023.slice/crio-2ccdb445e709a501be7b2d4cc3ef356db61070c396f2fa02da313cb15327dde8 WatchSource:0}: Error finding container 2ccdb445e709a501be7b2d4cc3ef356db61070c396f2fa02da313cb15327dde8: Status 404 returned error can't find the container with id 2ccdb445e709a501be7b2d4cc3ef356db61070c396f2fa02da313cb15327dde8 Oct 31 13:03:45.052494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.052465 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw"] Oct 31 13:03:45.052879 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:03:45.052857 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ead7707_8656_4809_82d6_d9f2f24581e8.slice/crio-56e9d3bb6c55fd44b647164fdb817010fdbc7e16634fce8192b4fce50aabf60f WatchSource:0}: Error finding container 56e9d3bb6c55fd44b647164fdb817010fdbc7e16634fce8192b4fce50aabf60f: Status 404 returned error can't find the container with id 56e9d3bb6c55fd44b647164fdb817010fdbc7e16634fce8192b4fce50aabf60f Oct 31 13:03:45.402419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.402333 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" event={"ID":"2ead7707-8656-4809-82d6-d9f2f24581e8","Type":"ContainerStarted","Data":"7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da"} Oct 31 13:03:45.402419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.402388 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" event={"ID":"2ead7707-8656-4809-82d6-d9f2f24581e8","Type":"ContainerStarted","Data":"56e9d3bb6c55fd44b647164fdb817010fdbc7e16634fce8192b4fce50aabf60f"} Oct 31 13:03:45.403292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.403274 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:45.405257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.405231 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" event={"ID":"277334af-3358-445e-96c5-8c6cb9b92023","Type":"ContainerStarted","Data":"0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e"} Oct 31 13:03:45.405384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.405279 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" event={"ID":"277334af-3358-445e-96c5-8c6cb9b92023","Type":"ContainerStarted","Data":"2ccdb445e709a501be7b2d4cc3ef356db61070c396f2fa02da313cb15327dde8"} Oct 31 13:03:45.405384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.405352 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:45.424505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.424455 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" podStartSLOduration=1.424442421 podStartE2EDuration="1.424442421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:45.422705161 +0000 UTC m=+1919.600423145" watchObservedRunningTime="2026-10-31 13:03:45.424442421 +0000 UTC m=+1919.602160405" Oct 31 13:03:45.445750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.445699 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" podStartSLOduration=1.445685326 podStartE2EDuration="1.445685326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:45.441617398 +0000 UTC m=+1919.619335382" watchObservedRunningTime="2026-10-31 13:03:45.445685326 +0000 UTC m=+1919.623403309" Oct 31 13:03:45.564452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.564413 5074 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:03:45.564629 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:45.564475 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 13:03:46.564286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:46.564251 5074 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:03:46.564715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:46.564303 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 13:03:47.564335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:47.564302 5074 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:03:47.564924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:47.564895 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 13:03:47.664932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:47.664898 5074 scope.go:122] "RemoveContainer" containerID="d4f683445e5d9b256a03d17b00dcaf4626ba60558008e93fbd66b0cdaf19d977" Oct 31 13:03:47.854529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:47.854262 5074 scope.go:122] "RemoveContainer" containerID="05b24aca2eb2a10a953ee4e1ae0dfff96b09611f4f91567bdc624c9c2a47bc99" Oct 31 13:03:47.993893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:47.993861 5074 scope.go:122] "RemoveContainer" containerID="d04a6adabf5fc7a2aef0eebc94e2613af6af7e450f68cf2acf98682a188d64c2" Oct 31 13:03:48.014203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:48.013781 5074 scope.go:122] "RemoveContainer" containerID="5163a82f95fd5c437f95d8d6ffc2821ace660d60cdb1a4f390705cecf5236f8a" Oct 31 13:03:48.054718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:48.054559 5074 scope.go:122] "RemoveContainer" containerID="33dde7601876dc04f5c47129a89a8ae38e7523b4594592027d31cbdabb43318e" Oct 31 13:03:48.563425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:48.563388 5074 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:03:48.563648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:48.563446 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 13:03:49.040724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:49.040685 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8"] Oct 31 13:03:49.045811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:49.045781 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a169d6be-j4xt8"] Oct 31 13:03:49.317294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:49.317222 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:03:49.564667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:49.563936 5074 patch_prober.go:29] interesting pod/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 13:03:49.564667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:49.563993 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 13:03:49.941752 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:49.941717 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c1356a-633f-42c4-8381-f8e5a6592f21" path="/var/lib/kubelet/pods/03c1356a-633f-42c4-8381-f8e5a6592f21/volumes" Oct 31 13:03:50.569033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:50.569003 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:50.569469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:50.569415 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:50.570310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:50.570287 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:03:53.367655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.360673 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh"] Oct 31 13:03:53.380660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.371996 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh"] Oct 31 13:03:53.380660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.372103 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.381260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.381077 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4989f9\"/\"default-dockercfg-xnq42\"" Oct 31 13:03:53.386678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.383105 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4989f9\"/\"openshift-service-ca.crt\"" Oct 31 13:03:53.386678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.383180 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4989f9\"/\"kube-root-ca.crt\"" Oct 31 13:03:53.388982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.388922 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4989f9\"/\"ipp-route-4989f9-kserve-self-signed-certs\"" Oct 31 13:03:53.558022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.557979 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e854b7-4ecb-45bd-a27c-b66143b75506-tls-certs\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.558285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.558260 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-dshm\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.558627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.558599 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-model-cache\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.558880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.558861 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-home\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.559101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.559085 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-tmp-dir\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.559373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.559352 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmstj\" (UniqueName: \"kubernetes.io/projected/a4e854b7-4ecb-45bd-a27c-b66143b75506-kube-api-access-rmstj\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.662774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.662679 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-dshm\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.663569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.663535 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-model-cache\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.663741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.663649 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-home\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.663741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.663711 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-tmp-dir\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.663876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.663856 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmstj\" (UniqueName: \"kubernetes.io/projected/a4e854b7-4ecb-45bd-a27c-b66143b75506-kube-api-access-rmstj\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.694082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.691464 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e854b7-4ecb-45bd-a27c-b66143b75506-tls-certs\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.694082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.664547 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-tmp-dir\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.694082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.691137 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-home\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.694082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.691400 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-dshm\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.694082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.664325 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-model-cache\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.702692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.702659 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e854b7-4ecb-45bd-a27c-b66143b75506-tls-certs\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.718934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.718901 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmstj\" (UniqueName: \"kubernetes.io/projected/a4e854b7-4ecb-45bd-a27c-b66143b75506-kube-api-access-rmstj\") pod \"ipp-route-4989f9-kserve-5dfd84797c-ph6lh\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:53.994463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:53.994408 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:54.547252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:54.547220 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh"] Oct 31 13:03:54.548921 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:03:54.548891 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e854b7_4ecb_45bd_a27c_b66143b75506.slice/crio-94bdef28abeb07cfbb0c3c24d2ad128368264de460bb716892b258c18c6e650c WatchSource:0}: Error finding container 94bdef28abeb07cfbb0c3c24d2ad128368264de460bb716892b258c18c6e650c: Status 404 returned error can't find the container with id 94bdef28abeb07cfbb0c3c24d2ad128368264de460bb716892b258c18c6e650c Oct 31 13:03:55.493159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:55.493097 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" event={"ID":"a4e854b7-4ecb-45bd-a27c-b66143b75506","Type":"ContainerStarted","Data":"f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832"} Oct 31 13:03:55.493159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:55.493153 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" event={"ID":"a4e854b7-4ecb-45bd-a27c-b66143b75506","Type":"ContainerStarted","Data":"94bdef28abeb07cfbb0c3c24d2ad128368264de460bb716892b258c18c6e650c"} Oct 31 13:03:55.520209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:55.520149 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" podStartSLOduration=2.520135207 podStartE2EDuration="2.520135207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:55.516401268 +0000 UTC m=+1929.694119272" watchObservedRunningTime="2026-10-31 13:03:55.520135207 +0000 UTC m=+1929.697853191" Oct 31 13:03:55.987596 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:03:55.987552 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:03:55.994930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:55.994902 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:03:56.420672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:56.420565 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:03:57.440822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:57.440340 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:03:59.934111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:59.934070 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc"] Oct 31 13:03:59.940829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:59.940796 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:03:59.955381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:59.955344 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e4420f-openshift-default-dockercfg-qxrj7\"" Oct 31 13:03:59.961264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:03:59.961233 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc"] Oct 31 13:04:00.093663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.075568 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-socket\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094060 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-podinfo\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094222 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-data\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094283 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istiod-ca-cert\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094455 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-credential-socket\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094496 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-token\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094522 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-certs\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094552 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-envoy\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.094665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.094622 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9psmr\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-kube-api-access-9psmr\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.199174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.199079 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-data\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.213209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.213171 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istiod-ca-cert\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.213579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.213558 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-credential-socket\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.213763 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.213745 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-token\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.213877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.213863 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-certs\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.213990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.213976 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-envoy\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.214143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.214117 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9psmr\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-kube-api-access-9psmr\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.229745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.214542 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-credential-socket\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.229932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.199868 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-data\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.229932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.215177 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istiod-ca-cert\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.229932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.215408 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-certs\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.230122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.229726 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-socket\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.230426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.230398 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-podinfo\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.231590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.231566 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-socket\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.242357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.242326 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-podinfo\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.269059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.269018 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-envoy\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.278732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.278660 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9psmr\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-kube-api-access-9psmr\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.291665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.288094 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-token\") pod \"e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.564650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.564605 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:00.894307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.894268 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc"] Oct 31 13:04:00.894527 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:04:00.894499 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28977cb_b60d_455e_a6f0_01f849d1ef9d.slice/crio-ac90dbe310f9718ce7ed3b3d397bc6e1567d2ec8ef595db551236eeb2f7862e7 WatchSource:0}: Error finding container ac90dbe310f9718ce7ed3b3d397bc6e1567d2ec8ef595db551236eeb2f7862e7: Status 404 returned error can't find the container with id ac90dbe310f9718ce7ed3b3d397bc6e1567d2ec8ef595db551236eeb2f7862e7 Oct 31 13:04:00.897705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.897669 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:00.897823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.897738 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:00.897823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:00.897784 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:01.555107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:01.555073 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" event={"ID":"a28977cb-b60d-455e-a6f0-01f849d1ef9d","Type":"ContainerStarted","Data":"05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1"} Oct 31 13:04:01.555503 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:01.555134 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" event={"ID":"a28977cb-b60d-455e-a6f0-01f849d1ef9d","Type":"ContainerStarted","Data":"ac90dbe310f9718ce7ed3b3d397bc6e1567d2ec8ef595db551236eeb2f7862e7"} Oct 31 13:04:01.565587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:01.565555 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:01.573477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:01.573445 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:01.593575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:01.589361 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" podStartSLOduration=2.589340371 podStartE2EDuration="2.589340371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:01.586189711 +0000 UTC m=+1935.763907696" watchObservedRunningTime="2026-10-31 13:04:01.589340371 +0000 UTC m=+1935.767058359" Oct 31 13:04:02.569802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:02.569397 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:02.571399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:02.571374 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:03.995384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:03.995347 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:04:04.014972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:04.014941 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:04:04.594491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:04.594461 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:04:05.048116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.048083 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d"] Oct 31 13:04:05.056722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.056678 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4677c662-79v4d"] Oct 31 13:04:05.433250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.433159 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp"] Oct 31 13:04:05.438899 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.438873 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.448790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.448762 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp"] Oct 31 13:04:05.470452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.470422 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e4420f-dockercfg-cg562\"" Oct 31 13:04:05.485225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.485198 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e4420f\"" Oct 31 13:04:05.538855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.538812 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.539240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.539217 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjk4k\" (UniqueName: \"kubernetes.io/projected/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-kube-api-access-wjk4k\") pod \"payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.640676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.640610 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.642012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.641983 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjk4k\" (UniqueName: \"kubernetes.io/projected/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-kube-api-access-wjk4k\") pod \"payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.643496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.641740 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.674667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.671958 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjk4k\" (UniqueName: \"kubernetes.io/projected/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-kube-api-access-wjk4k\") pod \"payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.762088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.762041 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:05.946143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:05.946105 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2f67af-fc90-40c8-873e-38b3f18fdc96" path="/var/lib/kubelet/pods/6b2f67af-fc90-40c8-873e-38b3f18fdc96/volumes" Oct 31 13:04:06.065904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:06.065861 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp"] Oct 31 13:04:06.612018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:06.611974 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" event={"ID":"aaabff29-68e0-4ed9-9b61-66ef2bdbf841","Type":"ContainerStarted","Data":"d6e9900c2bdfd977d6069c407f4c6957dbf75855c1e5fff58b7718e9f9eef511"} Oct 31 13:04:06.612206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:06.612086 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" event={"ID":"aaabff29-68e0-4ed9-9b61-66ef2bdbf841","Type":"ContainerStarted","Data":"d5c8094c9f20d6c7da39e0506bb875364aa7a371f2f53846ecb50091fc1338f6"} Oct 31 13:04:06.612301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:06.612226 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:06.634524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:06.634463 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" podStartSLOduration=1.634445227 podStartE2EDuration="1.634445227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:06.632188899 +0000 UTC m=+1940.809906902" watchObservedRunningTime="2026-10-31 13:04:06.634445227 +0000 UTC m=+1940.812163214" Oct 31 13:04:10.567455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:10.567363 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn"] Oct 31 13:04:14.333976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:14.333927 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:04:15.571656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:15.571127 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" podUID="a97874aa-95e6-4051-8117-e8f0ccaeeb42" containerName="payload-pre-processing" containerID="cri-o://12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f" gracePeriod=25 Oct 31 13:04:15.954446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:15.954407 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:04:16.096708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.096395 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97874aa-95e6-4051-8117-e8f0ccaeeb42-kube-api-access-6cx4c\" (UniqueName: \"kubernetes.io/projected/a97874aa-95e6-4051-8117-e8f0ccaeeb42-kube-api-access-6cx4c\") pod \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " Oct 31 13:04:16.096708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.096669 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97874aa-95e6-4051-8117-e8f0ccaeeb42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97874aa-95e6-4051-8117-e8f0ccaeeb42-plugins-config-volume\") pod \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\" (UID: \"a97874aa-95e6-4051-8117-e8f0ccaeeb42\") " Oct 31 13:04:16.131703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.124492 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97874aa-95e6-4051-8117-e8f0ccaeeb42-plugins-config-volume" pod "a97874aa-95e6-4051-8117-e8f0ccaeeb42" (UID: "a97874aa-95e6-4051-8117-e8f0ccaeeb42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:16.131982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.131949 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97874aa-95e6-4051-8117-e8f0ccaeeb42-kube-api-access-6cx4c" pod "a97874aa-95e6-4051-8117-e8f0ccaeeb42" (UID: "a97874aa-95e6-4051-8117-e8f0ccaeeb42"). InnerVolumeSpecName "kube-api-access-6cx4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:16.203847 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.203604 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97874aa-95e6-4051-8117-e8f0ccaeeb42-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:16.203847 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.203658 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cx4c\" (UniqueName: \"kubernetes.io/projected/a97874aa-95e6-4051-8117-e8f0ccaeeb42-kube-api-access-6cx4c\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:16.724504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.724462 5074 generic.go:356] "Generic (PLEG): container finished" podID="a97874aa-95e6-4051-8117-e8f0ccaeeb42" containerID="12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f" exitCode=0 Oct 31 13:04:16.725091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.724509 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" Oct 31 13:04:16.725091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.724508 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" event={"ID":"a97874aa-95e6-4051-8117-e8f0ccaeeb42","Type":"ContainerDied","Data":"12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f"} Oct 31 13:04:16.725091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.724643 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn" event={"ID":"a97874aa-95e6-4051-8117-e8f0ccaeeb42","Type":"ContainerDied","Data":"293619ed019b2afdfcb9133f4a051e842c946ba1306c57e5d2a93d48156f223e"} Oct 31 13:04:16.725091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.724674 5074 scope.go:122] "RemoveContainer" containerID="12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f" Oct 31 13:04:16.738620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.738073 5074 scope.go:122] "RemoveContainer" containerID="12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f" Oct 31 13:04:16.738620 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:04:16.738572 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f\": container with ID starting with 12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f not found: ID does not exist" containerID="12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f" Oct 31 13:04:16.738793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.738628 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f"} err="failed to get container status \"12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f\": rpc error: code = NotFound desc = could not find container \"12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f\": container with ID starting with 12600e3063c573ead329f3f1011c43e5eb24069418eea675aec204f5afc5526f not found: ID does not exist" Oct 31 13:04:16.783471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.783435 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn"] Oct 31 13:04:16.784204 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:16.784180 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c347ff-799c996c59-2n8sn"] Oct 31 13:04:17.625207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:17.625180 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:04:17.957788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:17.956317 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97874aa-95e6-4051-8117-e8f0ccaeeb42" path="/var/lib/kubelet/pods/a97874aa-95e6-4051-8117-e8f0ccaeeb42/volumes" Oct 31 13:04:25.995340 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:04:25.995298 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:04:29.250481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:29.250435 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf"] Oct 31 13:04:29.251103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:29.251070 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" containerID="cri-o://3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea" gracePeriod=30 Oct 31 13:04:34.719107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.719070 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:04:34.737478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737450 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-socket\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.737604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737498 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-certs\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.737604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737517 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-envoy\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.737604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737580 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-podinfo\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.737760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737625 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-data\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.737760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737718 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-socket" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.737872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737770 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-certs" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.737872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737843 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istiod-ca-cert\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.737995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737963 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-data" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.738073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.737976 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-kube-api-access-4fcr6\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-kube-api-access-4fcr6\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.738134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738102 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-token\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-token\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.738208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738176 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istiod-ca-cert" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:34.738263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738250 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-credential-socket\") pod \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\" (UID: \"dbfba18a-aab1-4e7b-ba83-55ead3e05acc\") " Oct 31 13:04:34.738588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738563 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-credential-socket" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.738883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738863 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.738957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738888 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.738957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738903 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.738957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738916 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.738957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.738929 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.740318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.740295 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-envoy" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:34.740377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.740295 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-podinfo" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:04:34.740530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.740511 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-token" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:34.740602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.740544 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-kube-api-access-4fcr6" pod "dbfba18a-aab1-4e7b-ba83-55ead3e05acc" (UID: "dbfba18a-aab1-4e7b-ba83-55ead3e05acc"). InnerVolumeSpecName "kube-api-access-4fcr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:34.850317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.850223 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcr6\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-kube-api-access-4fcr6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.850612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.850555 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.850612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.850579 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.850612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.850594 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfba18a-aab1-4e7b-ba83-55ead3e05acc-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:34.929308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.929274 5074 generic.go:356] "Generic (PLEG): container finished" podID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerID="3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea" exitCode=0 Oct 31 13:04:34.929456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.929324 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" Oct 31 13:04:34.929456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.929398 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" event={"ID":"dbfba18a-aab1-4e7b-ba83-55ead3e05acc","Type":"ContainerDied","Data":"3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea"} Oct 31 13:04:34.929549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.929466 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf" event={"ID":"dbfba18a-aab1-4e7b-ba83-55ead3e05acc","Type":"ContainerDied","Data":"27e1f20334e2d057b929d989ecdb0d29835438b4e7c0818c37b566fa159c2e37"} Oct 31 13:04:34.929549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.929491 5074 scope.go:122] "RemoveContainer" containerID="3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea" Oct 31 13:04:34.942141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.942119 5074 scope.go:122] "RemoveContainer" containerID="3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea" Oct 31 13:04:34.942386 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:04:34.942365 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea\": container with ID starting with 3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea not found: ID does not exist" containerID="3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea" Oct 31 13:04:34.942447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.942393 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea"} err="failed to get container status \"3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea\": rpc error: code = NotFound desc = could not find container \"3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea\": container with ID starting with 3fa0af461f4a04a0c4957c0238b9973cb228bf37fe36df3f5b512ebdc12a27ea not found: ID does not exist" Oct 31 13:04:34.955647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.955608 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf"] Oct 31 13:04:34.960277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:34.960256 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-49bffb-openshift-default-768b8587d5-vj9gf"] Oct 31 13:04:35.942494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:35.942462 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" path="/var/lib/kubelet/pods/dbfba18a-aab1-4e7b-ba83-55ead3e05acc/volumes" Oct 31 13:04:43.041429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.041393 5074 generic.go:356] "Generic (PLEG): container finished" podID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerID="05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1" exitCode=0 Oct 31 13:04:43.041888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.041465 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" event={"ID":"a28977cb-b60d-455e-a6f0-01f849d1ef9d","Type":"ContainerDied","Data":"05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1"} Oct 31 13:04:43.041939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.041913 5074 scope.go:122] "RemoveContainer" containerID="05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1" Oct 31 13:04:43.043559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.043534 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:43.043616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.043600 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:43.043683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.043665 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:04:43.556879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.556846 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw"] Oct 31 13:04:43.578658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:43.578607 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4"] Oct 31 13:04:44.063191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:44.062210 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" event={"ID":"a28977cb-b60d-455e-a6f0-01f849d1ef9d","Type":"ContainerStarted","Data":"2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd"} Oct 31 13:04:44.565151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:44.565110 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:44.566311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:44.566276 5074 patch_prober.go:29] interesting pod/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:04:44.566462 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:44.566357 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 13:04:45.565042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:45.564986 5074 patch_prober.go:29] interesting pod/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:04:45.565965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:45.565870 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 13:04:46.566314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:46.566260 5074 patch_prober.go:29] interesting pod/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:04:46.566893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:46.566336 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 13:04:47.565244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:47.565209 5074 patch_prober.go:29] interesting pod/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:04:47.565423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:47.565269 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 13:04:47.569180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:47.569152 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:48.153314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.153281 5074 scope.go:122] "RemoveContainer" containerID="fda75ed246acc13487c0c97d56d287cd2b520b7b10ede71b0dca8bc2d80892fe" Oct 31 13:04:48.163176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.163152 5074 scope.go:122] "RemoveContainer" containerID="de7f1c41d1ad2a9e873803fe6731277d014f037b46203dfae81e2ed91ffe0a65" Oct 31 13:04:48.207846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.207817 5074 scope.go:122] "RemoveContainer" containerID="eedbfc89b34f1c1f9afe77eb7db28f7f304a012b63b7f445ffd3271bd7e255eb" Oct 31 13:04:48.220518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.220492 5074 scope.go:122] "RemoveContainer" containerID="07450ab8a6ebc7f719521df156343d452f2afd4253efe497c29d1b56d51e8890" Oct 31 13:04:48.232727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.232027 5074 scope.go:122] "RemoveContainer" containerID="4083f2c390d15142737f5a8d637333eec13caf3be961c9db9bb25313819fe599" Oct 31 13:04:48.557676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.557617 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" podUID="2ead7707-8656-4809-82d6-d9f2f24581e8" containerName="payload-processing" containerID="cri-o://7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da" gracePeriod=25 Oct 31 13:04:48.565016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.564986 5074 patch_prober.go:29] interesting pod/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 13:04:48.565141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.565052 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 13:04:48.581239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.579656 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" podUID="277334af-3358-445e-96c5-8c6cb9b92023" containerName="payload-pre-processing" containerID="cri-o://0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e" gracePeriod=25 Oct 31 13:04:48.951254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:48.951210 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:04:49.102163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.102130 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:04:49.141651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.141605 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ead7707-8656-4809-82d6-d9f2f24581e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ead7707-8656-4809-82d6-d9f2f24581e8-plugins-config-volume\") pod \"2ead7707-8656-4809-82d6-d9f2f24581e8\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " Oct 31 13:04:49.141833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.141675 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ead7707-8656-4809-82d6-d9f2f24581e8-kube-api-access-vm7kg\" (UniqueName: \"kubernetes.io/projected/2ead7707-8656-4809-82d6-d9f2f24581e8-kube-api-access-vm7kg\") pod \"2ead7707-8656-4809-82d6-d9f2f24581e8\" (UID: \"2ead7707-8656-4809-82d6-d9f2f24581e8\") " Oct 31 13:04:49.142311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.142281 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ead7707-8656-4809-82d6-d9f2f24581e8-plugins-config-volume" pod "2ead7707-8656-4809-82d6-d9f2f24581e8" (UID: "2ead7707-8656-4809-82d6-d9f2f24581e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:49.142465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.142448 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ead7707-8656-4809-82d6-d9f2f24581e8-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:49.146734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.146682 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ead7707-8656-4809-82d6-d9f2f24581e8-kube-api-access-vm7kg" pod "2ead7707-8656-4809-82d6-d9f2f24581e8" (UID: "2ead7707-8656-4809-82d6-d9f2f24581e8"). InnerVolumeSpecName "kube-api-access-vm7kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:49.151582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.151555 5074 generic.go:356] "Generic (PLEG): container finished" podID="2ead7707-8656-4809-82d6-d9f2f24581e8" containerID="7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da" exitCode=0 Oct 31 13:04:49.151745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.151613 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" Oct 31 13:04:49.151811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.151740 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" event={"ID":"2ead7707-8656-4809-82d6-d9f2f24581e8","Type":"ContainerDied","Data":"7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da"} Oct 31 13:04:49.151811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.151787 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw" event={"ID":"2ead7707-8656-4809-82d6-d9f2f24581e8","Type":"ContainerDied","Data":"56e9d3bb6c55fd44b647164fdb817010fdbc7e16634fce8192b4fce50aabf60f"} Oct 31 13:04:49.151999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.151815 5074 scope.go:122] "RemoveContainer" containerID="7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da" Oct 31 13:04:49.154918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.154848 5074 generic.go:356] "Generic (PLEG): container finished" podID="277334af-3358-445e-96c5-8c6cb9b92023" containerID="0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e" exitCode=0 Oct 31 13:04:49.155053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.154914 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" event={"ID":"277334af-3358-445e-96c5-8c6cb9b92023","Type":"ContainerDied","Data":"0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e"} Oct 31 13:04:49.155053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.154969 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" event={"ID":"277334af-3358-445e-96c5-8c6cb9b92023","Type":"ContainerDied","Data":"2ccdb445e709a501be7b2d4cc3ef356db61070c396f2fa02da313cb15327dde8"} Oct 31 13:04:49.155053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.155030 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4" Oct 31 13:04:49.165935 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.165788 5074 scope.go:122] "RemoveContainer" containerID="7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da" Oct 31 13:04:49.166350 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:04:49.166217 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da\": container with ID starting with 7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da not found: ID does not exist" containerID="7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da" Oct 31 13:04:49.166350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.166252 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da"} err="failed to get container status \"7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da\": rpc error: code = NotFound desc = could not find container \"7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da\": container with ID starting with 7a1b25f39c1de74b107bd83645b22fc899956e6f4ab3eb86011216f7dd8964da not found: ID does not exist" Oct 31 13:04:49.166350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.166276 5074 scope.go:122] "RemoveContainer" containerID="0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e" Oct 31 13:04:49.193775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.193673 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw"] Oct 31 13:04:49.193775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.193733 5074 scope.go:122] "RemoveContainer" containerID="0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e" Oct 31 13:04:49.194138 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:04:49.194107 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e\": container with ID starting with 0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e not found: ID does not exist" containerID="0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e" Oct 31 13:04:49.194212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.194150 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e"} err="failed to get container status \"0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e\": rpc error: code = NotFound desc = could not find container \"0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e\": container with ID starting with 0ccc40863c369eb1e2d668ad962b8a4f361b8cfd87d005b20547c6ea9d31d53e not found: ID does not exist" Oct 31 13:04:49.194378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.194361 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa2079-745749fc47-6wlgw"] Oct 31 13:04:49.243687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.243652 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277334af-3358-445e-96c5-8c6cb9b92023-kube-api-access-9jdm7\" (UniqueName: \"kubernetes.io/projected/277334af-3358-445e-96c5-8c6cb9b92023-kube-api-access-9jdm7\") pod \"277334af-3358-445e-96c5-8c6cb9b92023\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " Oct 31 13:04:49.243833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.243747 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277334af-3358-445e-96c5-8c6cb9b92023-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277334af-3358-445e-96c5-8c6cb9b92023-plugins-config-volume\") pod \"277334af-3358-445e-96c5-8c6cb9b92023\" (UID: \"277334af-3358-445e-96c5-8c6cb9b92023\") " Oct 31 13:04:49.244139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.244110 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277334af-3358-445e-96c5-8c6cb9b92023-plugins-config-volume" pod "277334af-3358-445e-96c5-8c6cb9b92023" (UID: "277334af-3358-445e-96c5-8c6cb9b92023"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:49.244376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.244357 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm7kg\" (UniqueName: \"kubernetes.io/projected/2ead7707-8656-4809-82d6-d9f2f24581e8-kube-api-access-vm7kg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:49.244451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.244379 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277334af-3358-445e-96c5-8c6cb9b92023-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:49.245844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.245824 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277334af-3358-445e-96c5-8c6cb9b92023-kube-api-access-9jdm7" pod "277334af-3358-445e-96c5-8c6cb9b92023" (UID: "277334af-3358-445e-96c5-8c6cb9b92023"). InnerVolumeSpecName "kube-api-access-9jdm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:49.345303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.345264 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jdm7\" (UniqueName: \"kubernetes.io/projected/277334af-3358-445e-96c5-8c6cb9b92023-kube-api-access-9jdm7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:49.495652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.495604 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4"] Oct 31 13:04:49.499369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.499337 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa2079-6c9f497f95-bjbw4"] Oct 31 13:04:49.571438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.571405 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:49.572553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.572530 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:04:49.940921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.940848 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277334af-3358-445e-96c5-8c6cb9b92023" path="/var/lib/kubelet/pods/277334af-3358-445e-96c5-8c6cb9b92023/volumes" Oct 31 13:04:49.941308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:49.941292 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ead7707-8656-4809-82d6-d9f2f24581e8" path="/var/lib/kubelet/pods/2ead7707-8656-4809-82d6-d9f2f24581e8/volumes" Oct 31 13:04:55.985694 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:04:55.985653 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:04:56.036664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:56.036616 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t"] Oct 31 13:04:56.039787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:56.039764 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a1c0b-96t6t"] Oct 31 13:04:57.261052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:57.260709 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:04:57.997658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:57.989668 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1771e040-f7a3-44ee-8ae0-f460478a74bf" path="/var/lib/kubelet/pods/1771e040-f7a3-44ee-8ae0-f460478a74bf/volumes" Oct 31 13:04:59.503836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.503803 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4"] Oct 31 13:04:59.504400 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504385 5074 cpu_manager.go:414] "containerMap: removing container" podUID="277334af-3358-445e-96c5-8c6cb9b92023" containerName="payload-pre-processing" Oct 31 13:04:59.504443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504401 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277334af-3358-445e-96c5-8c6cb9b92023" containerName="payload-pre-processing" Oct 31 13:04:59.504443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504430 5074 cpu_manager.go:414] "containerMap: removing container" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" Oct 31 13:04:59.504443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504435 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" Oct 31 13:04:59.504443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504444 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2ead7707-8656-4809-82d6-d9f2f24581e8" containerName="payload-processing" Oct 31 13:04:59.504574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504449 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ead7707-8656-4809-82d6-d9f2f24581e8" containerName="payload-processing" Oct 31 13:04:59.504574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504472 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a97874aa-95e6-4051-8117-e8f0ccaeeb42" containerName="payload-pre-processing" Oct 31 13:04:59.504574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504477 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97874aa-95e6-4051-8117-e8f0ccaeeb42" containerName="payload-pre-processing" Oct 31 13:04:59.504574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504569 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfba18a-aab1-4e7b-ba83-55ead3e05acc" containerName="istio-proxy" Oct 31 13:04:59.504731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504579 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ead7707-8656-4809-82d6-d9f2f24581e8" containerName="payload-processing" Oct 31 13:04:59.504731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504590 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97874aa-95e6-4051-8117-e8f0ccaeeb42" containerName="payload-pre-processing" Oct 31 13:04:59.504731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.504597 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="277334af-3358-445e-96c5-8c6cb9b92023" containerName="payload-pre-processing" Oct 31 13:04:59.512403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.512377 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.518938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.518892 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:04:59.519092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.518989 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:04:59.519249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.519231 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:04:59.519906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.519865 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:04:59.524203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.524178 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4"] Oct 31 13:04:59.569914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.569870 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.570127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.570021 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.570127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.570089 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7tzb\" (UniqueName: \"kubernetes.io/projected/85f9d44c-2b1a-4265-9bc8-face74094904-kube-api-access-g7tzb\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.670924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.670894 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.671101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.670963 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.671101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.671027 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7tzb\" (UniqueName: \"kubernetes.io/projected/85f9d44c-2b1a-4265-9bc8-face74094904-kube-api-access-g7tzb\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.671681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.671655 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.673144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.673124 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.687504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.687476 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7tzb\" (UniqueName: \"kubernetes.io/projected/85f9d44c-2b1a-4265-9bc8-face74094904-kube-api-access-g7tzb\") pod \"maas-api-revoke-sub-f988d86e-ccxh4\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:04:59.829738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:04:59.829662 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:05:00.129467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:00.129438 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4"] Oct 31 13:05:00.130364 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:05:00.130334 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f9d44c_2b1a_4265_9bc8_face74094904.slice/crio-5a9aa60372d32d6ce3de168a8cdb55c8ff034a4a5594d892ab992dd3c5524f13 WatchSource:0}: Error finding container 5a9aa60372d32d6ce3de168a8cdb55c8ff034a4a5594d892ab992dd3c5524f13: Status 404 returned error can't find the container with id 5a9aa60372d32d6ce3de168a8cdb55c8ff034a4a5594d892ab992dd3c5524f13 Oct 31 13:05:00.306084 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:00.306043 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" event={"ID":"85f9d44c-2b1a-4265-9bc8-face74094904","Type":"ContainerStarted","Data":"5a9aa60372d32d6ce3de168a8cdb55c8ff034a4a5594d892ab992dd3c5524f13"} Oct 31 13:05:01.317508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:01.317469 5074 generic.go:356] "Generic (PLEG): container finished" podID="85f9d44c-2b1a-4265-9bc8-face74094904" containerID="0b3ef78e46f53c4a702849bac4bc16a9cbc0c02d8d2463ebdc41502b8413297f" exitCode=0 Oct 31 13:05:01.317947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:01.317571 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" event={"ID":"85f9d44c-2b1a-4265-9bc8-face74094904","Type":"ContainerDied","Data":"0b3ef78e46f53c4a702849bac4bc16a9cbc0c02d8d2463ebdc41502b8413297f"} Oct 31 13:05:02.560844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.560816 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:05:02.724965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.724931 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-xks-ca\") pod \"85f9d44c-2b1a-4265-9bc8-face74094904\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " Oct 31 13:05:02.725146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.725131 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-openshift-ca\") pod \"85f9d44c-2b1a-4265-9bc8-face74094904\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " Oct 31 13:05:02.725190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.725162 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f9d44c-2b1a-4265-9bc8-face74094904-kube-api-access-g7tzb\" (UniqueName: \"kubernetes.io/projected/85f9d44c-2b1a-4265-9bc8-face74094904-kube-api-access-g7tzb\") pod \"85f9d44c-2b1a-4265-9bc8-face74094904\" (UID: \"85f9d44c-2b1a-4265-9bc8-face74094904\") " Oct 31 13:05:02.725514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.725482 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-openshift-ca" pod "85f9d44c-2b1a-4265-9bc8-face74094904" (UID: "85f9d44c-2b1a-4265-9bc8-face74094904"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:02.725917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.725895 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.727142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.727113 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-xks-ca" pod "85f9d44c-2b1a-4265-9bc8-face74094904" (UID: "85f9d44c-2b1a-4265-9bc8-face74094904"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:02.727374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.727352 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f9d44c-2b1a-4265-9bc8-face74094904-kube-api-access-g7tzb" pod "85f9d44c-2b1a-4265-9bc8-face74094904" (UID: "85f9d44c-2b1a-4265-9bc8-face74094904"). InnerVolumeSpecName "kube-api-access-g7tzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:02.827045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.827007 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7tzb\" (UniqueName: \"kubernetes.io/projected/85f9d44c-2b1a-4265-9bc8-face74094904-kube-api-access-g7tzb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:02.827045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:02.827038 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f9d44c-2b1a-4265-9bc8-face74094904-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.336722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:03.336686 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" event={"ID":"85f9d44c-2b1a-4265-9bc8-face74094904","Type":"ContainerDied","Data":"5a9aa60372d32d6ce3de168a8cdb55c8ff034a4a5594d892ab992dd3c5524f13"} Oct 31 13:05:03.336901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:03.336746 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a9aa60372d32d6ce3de168a8cdb55c8ff034a4a5594d892ab992dd3c5524f13" Oct 31 13:05:03.336901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:03.336702 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4" Oct 31 13:05:08.122804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.122762 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh"] Oct 31 13:05:08.123578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.123555 5074 cpu_manager.go:414] "containerMap: removing container" podUID="85f9d44c-2b1a-4265-9bc8-face74094904" containerName="revoke-keys" Oct 31 13:05:08.123704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.123579 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f9d44c-2b1a-4265-9bc8-face74094904" containerName="revoke-keys" Oct 31 13:05:08.123829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.123813 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f9d44c-2b1a-4265-9bc8-face74094904" containerName="revoke-keys" Oct 31 13:05:08.129397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.129367 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.141198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.139555 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:05:08.141198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.139695 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:05:08.141694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.141481 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh"] Oct 31 13:05:08.141777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.141751 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:05:08.141899 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.141828 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:05:08.218879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.218575 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.218879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.218730 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqf8\" (UniqueName: \"kubernetes.io/projected/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-kube-api-access-qwqf8\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.218879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.218791 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.320604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.320571 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.320796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.320622 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqf8\" (UniqueName: \"kubernetes.io/projected/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-kube-api-access-qwqf8\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.320796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.320672 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.321355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.321331 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.323062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.323041 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.339179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.339154 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqf8\" (UniqueName: \"kubernetes.io/projected/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-kube-api-access-qwqf8\") pod \"maas-api-revoke-sub-88551bca-zcllh\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.455788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.455714 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:08.758654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:08.757124 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh"] Oct 31 13:05:08.767573 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:05:08.767532 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef22136a_2eb8_4ba2_b9c4_dc9e3efab3da.slice/crio-3f0c7693d796e8bfa8143d4e6fc4429b545947df1d5e526ecb4fd436537f7d6b WatchSource:0}: Error finding container 3f0c7693d796e8bfa8143d4e6fc4429b545947df1d5e526ecb4fd436537f7d6b: Status 404 returned error can't find the container with id 3f0c7693d796e8bfa8143d4e6fc4429b545947df1d5e526ecb4fd436537f7d6b Oct 31 13:05:09.398623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:09.398531 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" event={"ID":"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da","Type":"ContainerStarted","Data":"ca7534fd34dcea97d0551399c690b6d184d66f51d757f12a5b0c9269ace02ea2"} Oct 31 13:05:09.398623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:09.398596 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" event={"ID":"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da","Type":"ContainerStarted","Data":"3f0c7693d796e8bfa8143d4e6fc4429b545947df1d5e526ecb4fd436537f7d6b"} Oct 31 13:05:09.428031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:09.427979 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" podStartSLOduration=1.427964863 podStartE2EDuration="1.427964863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:09.427392367 +0000 UTC m=+2003.605110350" watchObservedRunningTime="2026-10-31 13:05:09.427964863 +0000 UTC m=+2003.605682846" Oct 31 13:05:10.409535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:10.409497 5074 generic.go:356] "Generic (PLEG): container finished" podID="ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" containerID="ca7534fd34dcea97d0551399c690b6d184d66f51d757f12a5b0c9269ace02ea2" exitCode=0 Oct 31 13:05:10.409972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:10.409581 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" event={"ID":"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da","Type":"ContainerDied","Data":"ca7534fd34dcea97d0551399c690b6d184d66f51d757f12a5b0c9269ace02ea2"} Oct 31 13:05:11.594519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.594490 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:11.710839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.710757 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-openshift-ca\") pod \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " Oct 31 13:05:11.710839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.710836 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-kube-api-access-qwqf8\" (UniqueName: \"kubernetes.io/projected/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-kube-api-access-qwqf8\") pod \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " Oct 31 13:05:11.711086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.711028 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-xks-ca\") pod \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\" (UID: \"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da\") " Oct 31 13:05:11.711248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.711218 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-openshift-ca" pod "ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" (UID: "ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:11.711705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.711681 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:11.713030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.713003 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-xks-ca" pod "ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" (UID: "ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:11.713120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.713087 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-kube-api-access-qwqf8" pod "ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" (UID: "ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da"). InnerVolumeSpecName "kube-api-access-qwqf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:11.812985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.812951 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:11.812985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:11.812982 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwqf8\" (UniqueName: \"kubernetes.io/projected/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da-kube-api-access-qwqf8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:12.020125 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:05:12.020089 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef22136a_2eb8_4ba2_b9c4_dc9e3efab3da.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:05:12.376196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.376108 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4"] Oct 31 13:05:12.377158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.377133 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" containerName="revoke-keys" Oct 31 13:05:12.377158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.377157 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" containerName="revoke-keys" Oct 31 13:05:12.377386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.377369 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" containerName="revoke-keys" Oct 31 13:05:12.395829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.395799 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4"] Oct 31 13:05:12.396139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.396123 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.449159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.449129 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" event={"ID":"ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da","Type":"ContainerDied","Data":"3f0c7693d796e8bfa8143d4e6fc4429b545947df1d5e526ecb4fd436537f7d6b"} Oct 31 13:05:12.449337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.449327 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f0c7693d796e8bfa8143d4e6fc4429b545947df1d5e526ecb4fd436537f7d6b" Oct 31 13:05:12.449457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.449448 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh" Oct 31 13:05:12.529683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.529620 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.529901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.529777 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.529901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.529802 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpsqz\" (UniqueName: \"kubernetes.io/projected/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-kube-api-access-lpsqz\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.646701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.643820 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.646701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.643956 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.646701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.643990 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpsqz\" (UniqueName: \"kubernetes.io/projected/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-kube-api-access-lpsqz\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.646701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.645548 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.652170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.650078 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.694656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.692374 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpsqz\" (UniqueName: \"kubernetes.io/projected/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-kube-api-access-lpsqz\") pod \"maas-api-revoke-sub-dc881831-fbcv4\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:12.717907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:12.717870 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:13.076572 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:13.074425 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4"] Oct 31 13:05:13.629415 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:13.629369 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" event={"ID":"59c4c5db-ef96-405d-9c13-dc90e6bc17e3","Type":"ContainerStarted","Data":"13f9ffd35dc449d9298384bf7ed440c4f25e9d5c654c541e01d9c1c5a2ead2d7"} Oct 31 13:05:14.639309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:14.639273 5074 generic.go:356] "Generic (PLEG): container finished" podID="59c4c5db-ef96-405d-9c13-dc90e6bc17e3" containerID="8f0d012593ef79354071cc6756da8fd037f07d71ea1c42f8ff774a3175c138bd" exitCode=0 Oct 31 13:05:14.639839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:14.639624 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" event={"ID":"59c4c5db-ef96-405d-9c13-dc90e6bc17e3","Type":"ContainerDied","Data":"8f0d012593ef79354071cc6756da8fd037f07d71ea1c42f8ff774a3175c138bd"} Oct 31 13:05:15.916445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:15.916379 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:16.021658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.021296 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-kube-api-access-lpsqz\" (UniqueName: \"kubernetes.io/projected/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-kube-api-access-lpsqz\") pod \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " Oct 31 13:05:16.021658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.021379 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-xks-ca\") pod \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " Oct 31 13:05:16.021658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.021434 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-openshift-ca\") pod \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\" (UID: \"59c4c5db-ef96-405d-9c13-dc90e6bc17e3\") " Oct 31 13:05:16.038662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.037382 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-openshift-ca" pod "59c4c5db-ef96-405d-9c13-dc90e6bc17e3" (UID: "59c4c5db-ef96-405d-9c13-dc90e6bc17e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:16.056020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.054916 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-kube-api-access-lpsqz" pod "59c4c5db-ef96-405d-9c13-dc90e6bc17e3" (UID: "59c4c5db-ef96-405d-9c13-dc90e6bc17e3"). InnerVolumeSpecName "kube-api-access-lpsqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:16.056748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.056693 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-xks-ca" pod "59c4c5db-ef96-405d-9c13-dc90e6bc17e3" (UID: "59c4c5db-ef96-405d-9c13-dc90e6bc17e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:16.124303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.124260 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpsqz\" (UniqueName: \"kubernetes.io/projected/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-kube-api-access-lpsqz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:16.124303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.124302 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:16.124543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.124319 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c4c5db-ef96-405d-9c13-dc90e6bc17e3-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:16.670456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.670419 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" Oct 31 13:05:16.670456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.670443 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4" event={"ID":"59c4c5db-ef96-405d-9c13-dc90e6bc17e3","Type":"ContainerDied","Data":"13f9ffd35dc449d9298384bf7ed440c4f25e9d5c654c541e01d9c1c5a2ead2d7"} Oct 31 13:05:16.670721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:16.670501 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13f9ffd35dc449d9298384bf7ed440c4f25e9d5c654c541e01d9c1c5a2ead2d7" Oct 31 13:05:21.292760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.292726 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66"] Oct 31 13:05:21.293582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.293559 5074 cpu_manager.go:414] "containerMap: removing container" podUID="59c4c5db-ef96-405d-9c13-dc90e6bc17e3" containerName="revoke-keys" Oct 31 13:05:21.293582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.293581 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c4c5db-ef96-405d-9c13-dc90e6bc17e3" containerName="revoke-keys" Oct 31 13:05:21.293804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.293789 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c4c5db-ef96-405d-9c13-dc90e6bc17e3" containerName="revoke-keys" Oct 31 13:05:21.297849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.297816 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.307658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.307614 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:05:21.308284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.308129 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:05:21.312999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.311955 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66"] Oct 31 13:05:21.377274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.377240 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:05:21.377443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.377424 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:05:21.416153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.416121 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.416302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.416248 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf4p\" (UniqueName: \"kubernetes.io/projected/67842144-cce9-4859-b982-d3fea1b70acf-kube-api-access-dvf4p\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.416302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.416286 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.517914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.517883 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.518130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.517951 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf4p\" (UniqueName: \"kubernetes.io/projected/67842144-cce9-4859-b982-d3fea1b70acf-kube-api-access-dvf4p\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.518130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.517998 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.518727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.518707 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.520342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.520319 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.534898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.534873 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf4p\" (UniqueName: \"kubernetes.io/projected/67842144-cce9-4859-b982-d3fea1b70acf-kube-api-access-dvf4p\") pod \"maas-api-revoke-sub-0ea7da3d-ljq66\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.623468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.623395 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:21.917791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:21.917765 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66"] Oct 31 13:05:21.918757 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:05:21.918730 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67842144_cce9_4859_b982_d3fea1b70acf.slice/crio-9de2d7eb5b152afc19a5f30a116c773ff12f100bd1f55fe7b7079ad7f4d6ce62 WatchSource:0}: Error finding container 9de2d7eb5b152afc19a5f30a116c773ff12f100bd1f55fe7b7079ad7f4d6ce62: Status 404 returned error can't find the container with id 9de2d7eb5b152afc19a5f30a116c773ff12f100bd1f55fe7b7079ad7f4d6ce62 Oct 31 13:05:22.823253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:22.823209 5074 generic.go:356] "Generic (PLEG): container finished" podID="67842144-cce9-4859-b982-d3fea1b70acf" containerID="4519fa93a3d3caee1e89f8558143c720372b9a530167abafb2ddb2b0b77a3f7d" exitCode=0 Oct 31 13:05:22.823698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:22.823293 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" event={"ID":"67842144-cce9-4859-b982-d3fea1b70acf","Type":"ContainerDied","Data":"4519fa93a3d3caee1e89f8558143c720372b9a530167abafb2ddb2b0b77a3f7d"} Oct 31 13:05:22.823698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:22.823364 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" event={"ID":"67842144-cce9-4859-b982-d3fea1b70acf","Type":"ContainerStarted","Data":"9de2d7eb5b152afc19a5f30a116c773ff12f100bd1f55fe7b7079ad7f4d6ce62"} Oct 31 13:05:24.019209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.019184 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:24.052419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.052390 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-openshift-ca\") pod \"67842144-cce9-4859-b982-d3fea1b70acf\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " Oct 31 13:05:24.052594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.052477 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-xks-ca\") pod \"67842144-cce9-4859-b982-d3fea1b70acf\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " Oct 31 13:05:24.052714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.052693 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67842144-cce9-4859-b982-d3fea1b70acf-kube-api-access-dvf4p\" (UniqueName: \"kubernetes.io/projected/67842144-cce9-4859-b982-d3fea1b70acf-kube-api-access-dvf4p\") pod \"67842144-cce9-4859-b982-d3fea1b70acf\" (UID: \"67842144-cce9-4859-b982-d3fea1b70acf\") " Oct 31 13:05:24.052814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.052791 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-openshift-ca" pod "67842144-cce9-4859-b982-d3fea1b70acf" (UID: "67842144-cce9-4859-b982-d3fea1b70acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:24.053481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.053451 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:24.055304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.055281 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-xks-ca" pod "67842144-cce9-4859-b982-d3fea1b70acf" (UID: "67842144-cce9-4859-b982-d3fea1b70acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:24.055471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.055448 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67842144-cce9-4859-b982-d3fea1b70acf-kube-api-access-dvf4p" pod "67842144-cce9-4859-b982-d3fea1b70acf" (UID: "67842144-cce9-4859-b982-d3fea1b70acf"). InnerVolumeSpecName "kube-api-access-dvf4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:24.155195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.155109 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvf4p\" (UniqueName: \"kubernetes.io/projected/67842144-cce9-4859-b982-d3fea1b70acf-kube-api-access-dvf4p\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:24.155195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.155140 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67842144-cce9-4859-b982-d3fea1b70acf-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:24.840628 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.840597 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" Oct 31 13:05:24.840838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.840620 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66" event={"ID":"67842144-cce9-4859-b982-d3fea1b70acf","Type":"ContainerDied","Data":"9de2d7eb5b152afc19a5f30a116c773ff12f100bd1f55fe7b7079ad7f4d6ce62"} Oct 31 13:05:24.840838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:24.840713 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de2d7eb5b152afc19a5f30a116c773ff12f100bd1f55fe7b7079ad7f4d6ce62" Oct 31 13:05:25.482235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.482198 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf"] Oct 31 13:05:25.482836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.482820 5074 cpu_manager.go:414] "containerMap: removing container" podUID="67842144-cce9-4859-b982-d3fea1b70acf" containerName="revoke-keys" Oct 31 13:05:25.482881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.482837 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67842144-cce9-4859-b982-d3fea1b70acf" containerName="revoke-keys" Oct 31 13:05:25.482984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.482975 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="67842144-cce9-4859-b982-d3fea1b70acf" containerName="revoke-keys" Oct 31 13:05:25.486509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.486492 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.496453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.496430 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:05:25.496616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.496453 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:05:25.496616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.496500 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:05:25.496991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.496534 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:05:25.498478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.498452 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf"] Oct 31 13:05:25.582603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.582564 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.582811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.582613 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.582811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.582662 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp8s\" (UniqueName: \"kubernetes.io/projected/af15212e-f3f9-486e-b282-1fde1ca23067-kube-api-access-xfp8s\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.684226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.684188 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.684226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.684226 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.684445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.684254 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp8s\" (UniqueName: \"kubernetes.io/projected/af15212e-f3f9-486e-b282-1fde1ca23067-kube-api-access-xfp8s\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.684829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.684804 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.686664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.686628 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.697151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.697123 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp8s\" (UniqueName: \"kubernetes.io/projected/af15212e-f3f9-486e-b282-1fde1ca23067-kube-api-access-xfp8s\") pod \"maas-api-revoke-sub-e8b278de-4n6qf\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.803017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:25.802946 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:25.985880 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:05:25.985835 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:05:26.131294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:26.131200 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf"] Oct 31 13:05:26.876340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:26.876300 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" event={"ID":"af15212e-f3f9-486e-b282-1fde1ca23067","Type":"ContainerStarted","Data":"d3bad3ab37471fca1a95991614312bf345544f9ccc904c7e7046871f0ff53460"} Oct 31 13:05:26.876883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:26.876367 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" event={"ID":"af15212e-f3f9-486e-b282-1fde1ca23067","Type":"ContainerStarted","Data":"50632639d276cf8bae0d442d267c5422cd5fb32e7828c3e6afd0711ca1b3ae7e"} Oct 31 13:05:27.886439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:27.886405 5074 generic.go:356] "Generic (PLEG): container finished" podID="af15212e-f3f9-486e-b282-1fde1ca23067" containerID="d3bad3ab37471fca1a95991614312bf345544f9ccc904c7e7046871f0ff53460" exitCode=0 Oct 31 13:05:27.886942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:27.886478 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" event={"ID":"af15212e-f3f9-486e-b282-1fde1ca23067","Type":"ContainerDied","Data":"d3bad3ab37471fca1a95991614312bf345544f9ccc904c7e7046871f0ff53460"} Oct 31 13:05:28.037265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:28.037228 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5"] Oct 31 13:05:28.040528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:28.040505 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e827e56-66kw5"] Oct 31 13:05:29.056823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.056800 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:29.148918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.148844 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-openshift-ca\") pod \"af15212e-f3f9-486e-b282-1fde1ca23067\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " Oct 31 13:05:29.149073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.148968 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-xks-ca\") pod \"af15212e-f3f9-486e-b282-1fde1ca23067\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " Oct 31 13:05:29.149127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.149083 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af15212e-f3f9-486e-b282-1fde1ca23067-kube-api-access-xfp8s\" (UniqueName: \"kubernetes.io/projected/af15212e-f3f9-486e-b282-1fde1ca23067-kube-api-access-xfp8s\") pod \"af15212e-f3f9-486e-b282-1fde1ca23067\" (UID: \"af15212e-f3f9-486e-b282-1fde1ca23067\") " Oct 31 13:05:29.149200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.149171 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-openshift-ca" pod "af15212e-f3f9-486e-b282-1fde1ca23067" (UID: "af15212e-f3f9-486e-b282-1fde1ca23067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:29.149507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.149489 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:29.151009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.150983 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-xks-ca" pod "af15212e-f3f9-486e-b282-1fde1ca23067" (UID: "af15212e-f3f9-486e-b282-1fde1ca23067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:29.151135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.151118 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af15212e-f3f9-486e-b282-1fde1ca23067-kube-api-access-xfp8s" pod "af15212e-f3f9-486e-b282-1fde1ca23067" (UID: "af15212e-f3f9-486e-b282-1fde1ca23067"). InnerVolumeSpecName "kube-api-access-xfp8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:29.250362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.250327 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af15212e-f3f9-486e-b282-1fde1ca23067-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:29.250362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.250361 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfp8s\" (UniqueName: \"kubernetes.io/projected/af15212e-f3f9-486e-b282-1fde1ca23067-kube-api-access-xfp8s\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:29.999662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.995461 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" Oct 31 13:05:29.999662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.996002 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a9e022-f6ac-4689-8124-e00b722ebd3f" path="/var/lib/kubelet/pods/05a9e022-f6ac-4689-8124-e00b722ebd3f/volumes" Oct 31 13:05:29.999662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.996664 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf" event={"ID":"af15212e-f3f9-486e-b282-1fde1ca23067","Type":"ContainerDied","Data":"50632639d276cf8bae0d442d267c5422cd5fb32e7828c3e6afd0711ca1b3ae7e"} Oct 31 13:05:29.999662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:29.996708 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50632639d276cf8bae0d442d267c5422cd5fb32e7828c3e6afd0711ca1b3ae7e" Oct 31 13:05:33.626086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.626048 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph"] Oct 31 13:05:33.626660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.626628 5074 cpu_manager.go:414] "containerMap: removing container" podUID="af15212e-f3f9-486e-b282-1fde1ca23067" containerName="revoke-keys" Oct 31 13:05:33.626660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.626657 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af15212e-f3f9-486e-b282-1fde1ca23067" containerName="revoke-keys" Oct 31 13:05:33.626789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.626780 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="af15212e-f3f9-486e-b282-1fde1ca23067" containerName="revoke-keys" Oct 31 13:05:33.630620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.630603 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.635082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.635053 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:05:33.636489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.636451 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:05:33.636489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.636454 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:05:33.636677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.636496 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:05:33.641113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.641094 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph"] Oct 31 13:05:33.746431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.746391 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.746607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.746444 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.746607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.746564 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l5lt\" (UniqueName: \"kubernetes.io/projected/ef5c5663-bf99-446d-bf67-33498402b507-kube-api-access-6l5lt\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.850456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.849192 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.850456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.849264 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.850456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.849351 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5lt\" (UniqueName: \"kubernetes.io/projected/ef5c5663-bf99-446d-bf67-33498402b507-kube-api-access-6l5lt\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.850839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.850660 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.868514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.868484 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.870582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.870551 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5lt\" (UniqueName: \"kubernetes.io/projected/ef5c5663-bf99-446d-bf67-33498402b507-kube-api-access-6l5lt\") pod \"maas-api-revoke-sub-92f41a62-d9xph\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:33.946683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:33.946585 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:34.243169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:34.243138 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph"] Oct 31 13:05:34.246960 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:05:34.246930 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5c5663_bf99_446d_bf67_33498402b507.slice/crio-46218e658af10ee73edd5e6eacd1cc074279a8ab5084d66cbcecad94164c39f2 WatchSource:0}: Error finding container 46218e658af10ee73edd5e6eacd1cc074279a8ab5084d66cbcecad94164c39f2: Status 404 returned error can't find the container with id 46218e658af10ee73edd5e6eacd1cc074279a8ab5084d66cbcecad94164c39f2 Oct 31 13:05:35.038598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.038562 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" event={"ID":"ef5c5663-bf99-446d-bf67-33498402b507","Type":"ContainerStarted","Data":"da79e2dafcb0ac4d71d1b36c89ec6e39ab9d950a8cfb791194453b1fb2a62e90"} Oct 31 13:05:35.039008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.038617 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" event={"ID":"ef5c5663-bf99-446d-bf67-33498402b507","Type":"ContainerStarted","Data":"46218e658af10ee73edd5e6eacd1cc074279a8ab5084d66cbcecad94164c39f2"} Oct 31 13:05:35.059151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.059093 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" podStartSLOduration=2.059080306 podStartE2EDuration="2.059080306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:35.05808248 +0000 UTC m=+2029.235800465" watchObservedRunningTime="2026-10-31 13:05:35.059080306 +0000 UTC m=+2029.236798345" Oct 31 13:05:35.311618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.311525 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:05:35.767650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.767600 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c"] Oct 31 13:05:35.771984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.771962 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.806124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.806082 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c"] Oct 31 13:05:35.878622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.878588 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.878871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.878735 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwfc\" (UniqueName: \"kubernetes.io/projected/f9eeabb9-f211-49b3-b24f-7c9b3855409a-kube-api-access-xlwfc\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.878871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.878801 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.980772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.980733 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.980968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.980795 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwfc\" (UniqueName: \"kubernetes.io/projected/f9eeabb9-f211-49b3-b24f-7c9b3855409a-kube-api-access-xlwfc\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.980968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.980855 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.981451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.981423 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:35.983237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:35.983202 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:36.002929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:36.002904 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwfc\" (UniqueName: \"kubernetes.io/projected/f9eeabb9-f211-49b3-b24f-7c9b3855409a-kube-api-access-xlwfc\") pod \"maas-api-revoke-sub-27b60d0b-k5g7c\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:36.047105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:36.047044 5074 generic.go:356] "Generic (PLEG): container finished" podID="ef5c5663-bf99-446d-bf67-33498402b507" containerID="da79e2dafcb0ac4d71d1b36c89ec6e39ab9d950a8cfb791194453b1fb2a62e90" exitCode=0 Oct 31 13:05:36.047422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:36.047122 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" event={"ID":"ef5c5663-bf99-446d-bf67-33498402b507","Type":"ContainerDied","Data":"da79e2dafcb0ac4d71d1b36c89ec6e39ab9d950a8cfb791194453b1fb2a62e90"} Oct 31 13:05:36.085476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:36.085438 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:37.097660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:37.097210 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c"] Oct 31 13:05:38.222287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.221810 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:38.299164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.266305 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:38.321658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.307797 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:38.561394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.561347 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:38.640717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.640475 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:38.812984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.812940 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" event={"ID":"f9eeabb9-f211-49b3-b24f-7c9b3855409a","Type":"ContainerStarted","Data":"0e24f30f44b1e792044cbe73d19a01d8ea4daebddad2284dbda4413c5e1becf3"} Oct 31 13:05:38.837304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.837269 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:38.862274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.862221 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:38.992662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:38.979049 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:39.028658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.027975 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-openshift-ca\") pod \"ef5c5663-bf99-446d-bf67-33498402b507\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " Oct 31 13:05:39.033316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.033290 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5c5663-bf99-446d-bf67-33498402b507-kube-api-access-6l5lt\" (UniqueName: \"kubernetes.io/projected/ef5c5663-bf99-446d-bf67-33498402b507-kube-api-access-6l5lt\") pod \"ef5c5663-bf99-446d-bf67-33498402b507\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " Oct 31 13:05:39.033859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.033838 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-xks-ca\") pod \"ef5c5663-bf99-446d-bf67-33498402b507\" (UID: \"ef5c5663-bf99-446d-bf67-33498402b507\") " Oct 31 13:05:39.035081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.030606 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-openshift-ca" pod "ef5c5663-bf99-446d-bf67-33498402b507" (UID: "ef5c5663-bf99-446d-bf67-33498402b507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:39.163821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.141812 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:39.231696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.203114 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:39.251486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.251439 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 13:05:39.270392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.270323 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:39.271293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.271274 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:05:39.351653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.339826 5074 patch_prober.go:29] interesting pod/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:05:39.351653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.339882 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:39.361151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.360968 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-xks-ca" pod "ef5c5663-bf99-446d-bf67-33498402b507" (UID: "ef5c5663-bf99-446d-bf67-33498402b507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:39.361151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.361113 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5c5663-bf99-446d-bf67-33498402b507-kube-api-access-6l5lt" pod "ef5c5663-bf99-446d-bf67-33498402b507" (UID: "ef5c5663-bf99-446d-bf67-33498402b507"). InnerVolumeSpecName "kube-api-access-6l5lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:39.368656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.363679 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:05:39.445246 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.445205 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l5lt\" (UniqueName: \"kubernetes.io/projected/ef5c5663-bf99-446d-bf67-33498402b507-kube-api-access-6l5lt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:39.450255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.450226 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c5663-bf99-446d-bf67-33498402b507-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:39.583771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.583724 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 13:05:39.584862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.584836 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 13:05:39.716404 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.716022 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:39.816682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.774745 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 13:05:39.816682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.775170 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 13:05:39.914717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.895879 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 13:05:39.931492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.931258 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 13:05:39.947686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:39.931905 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:40.040503 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.040411 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:40.231665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.227709 5074 generic.go:356] "Generic (PLEG): container finished" podID="f9eeabb9-f211-49b3-b24f-7c9b3855409a" containerID="0c887cee4793cf060b5ec63658e4ab9b29b7008378006e6b837515b74295ed39" exitCode=0 Oct 31 13:05:40.232491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.232385 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" Oct 31 13:05:40.235319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.233594 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" event={"ID":"f9eeabb9-f211-49b3-b24f-7c9b3855409a","Type":"ContainerDied","Data":"0c887cee4793cf060b5ec63658e4ab9b29b7008378006e6b837515b74295ed39"} Oct 31 13:05:40.235319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.233697 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 13:05:40.235319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.233719 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph" event={"ID":"ef5c5663-bf99-446d-bf67-33498402b507","Type":"ContainerDied","Data":"46218e658af10ee73edd5e6eacd1cc074279a8ab5084d66cbcecad94164c39f2"} Oct 31 13:05:40.235319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.233746 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46218e658af10ee73edd5e6eacd1cc074279a8ab5084d66cbcecad94164c39f2" Oct 31 13:05:40.235319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.233780 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 13:05:40.235319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.233813 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 13:05:40.265147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.265093 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:40.338978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.338818 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:05:40.380042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.380014 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 13:05:40.404652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:40.403291 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 13:05:41.441862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.441838 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:41.517520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.517437 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9eeabb9-f211-49b3-b24f-7c9b3855409a-kube-api-access-xlwfc\" (UniqueName: \"kubernetes.io/projected/f9eeabb9-f211-49b3-b24f-7c9b3855409a-kube-api-access-xlwfc\") pod \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " Oct 31 13:05:41.517694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.517550 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-xks-ca\") pod \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " Oct 31 13:05:41.517778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.517689 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-openshift-ca\") pod \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\" (UID: \"f9eeabb9-f211-49b3-b24f-7c9b3855409a\") " Oct 31 13:05:41.518758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.518727 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-openshift-ca" pod "f9eeabb9-f211-49b3-b24f-7c9b3855409a" (UID: "f9eeabb9-f211-49b3-b24f-7c9b3855409a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:41.520648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.520590 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9eeabb9-f211-49b3-b24f-7c9b3855409a-kube-api-access-xlwfc" pod "f9eeabb9-f211-49b3-b24f-7c9b3855409a" (UID: "f9eeabb9-f211-49b3-b24f-7c9b3855409a"). InnerVolumeSpecName "kube-api-access-xlwfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:41.536894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.532797 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-xks-ca" pod "f9eeabb9-f211-49b3-b24f-7c9b3855409a" (UID: "f9eeabb9-f211-49b3-b24f-7c9b3855409a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:41.619478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.619444 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlwfc\" (UniqueName: \"kubernetes.io/projected/f9eeabb9-f211-49b3-b24f-7c9b3855409a-kube-api-access-xlwfc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:41.619478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.619472 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:41.619478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:41.619483 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9eeabb9-f211-49b3-b24f-7c9b3855409a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.249081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:42.249046 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" Oct 31 13:05:42.249234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:42.249083 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c" event={"ID":"f9eeabb9-f211-49b3-b24f-7c9b3855409a","Type":"ContainerDied","Data":"0e24f30f44b1e792044cbe73d19a01d8ea4daebddad2284dbda4413c5e1becf3"} Oct 31 13:05:42.249234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:42.249128 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e24f30f44b1e792044cbe73d19a01d8ea4daebddad2284dbda4413c5e1becf3" Oct 31 13:05:48.411386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.408244 5074 scope.go:122] "RemoveContainer" containerID="3d0802d4280112f499df31c3d0d317455f2a463a782fc6a2945ed601fb7ca29f" Oct 31 13:05:48.512035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.511999 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9"] Oct 31 13:05:48.512663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.512625 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ef5c5663-bf99-446d-bf67-33498402b507" containerName="revoke-keys" Oct 31 13:05:48.512663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.512653 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5c5663-bf99-446d-bf67-33498402b507" containerName="revoke-keys" Oct 31 13:05:48.512773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.512666 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f9eeabb9-f211-49b3-b24f-7c9b3855409a" containerName="revoke-keys" Oct 31 13:05:48.512773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.512672 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9eeabb9-f211-49b3-b24f-7c9b3855409a" containerName="revoke-keys" Oct 31 13:05:48.512773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.512757 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5c5663-bf99-446d-bf67-33498402b507" containerName="revoke-keys" Oct 31 13:05:48.512773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.512767 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9eeabb9-f211-49b3-b24f-7c9b3855409a" containerName="revoke-keys" Oct 31 13:05:48.518102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.518083 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.526688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.526615 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:05:48.526808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.526717 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:05:48.526808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.526799 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:05:48.529133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.529109 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:05:48.533233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.533210 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9"] Oct 31 13:05:48.666544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.665892 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.666544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.666040 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqf8\" (UniqueName: \"kubernetes.io/projected/79eeec8b-e321-471c-9e62-722bc9ba45af-kube-api-access-fhqf8\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.666544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.666236 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.767237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.767201 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.767408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.767280 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqf8\" (UniqueName: \"kubernetes.io/projected/79eeec8b-e321-471c-9e62-722bc9ba45af-kube-api-access-fhqf8\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.767491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.767445 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.768329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.768299 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.769606 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.769580 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.782259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.782234 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqf8\" (UniqueName: \"kubernetes.io/projected/79eeec8b-e321-471c-9e62-722bc9ba45af-kube-api-access-fhqf8\") pod \"maas-api-revoke-sub-dcfa247b-smxw9\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:48.837842 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:48.837816 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:49.173774 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:05:49.173742 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79eeec8b_e321_471c_9e62_722bc9ba45af.slice/crio-a801f852557000d4bda350a1aca84f3c7968248e749a51fc2da02fa734f6e10c WatchSource:0}: Error finding container a801f852557000d4bda350a1aca84f3c7968248e749a51fc2da02fa734f6e10c: Status 404 returned error can't find the container with id a801f852557000d4bda350a1aca84f3c7968248e749a51fc2da02fa734f6e10c Oct 31 13:05:49.173931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.173867 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9"] Oct 31 13:05:49.306188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.306128 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" event={"ID":"79eeec8b-e321-471c-9e62-722bc9ba45af","Type":"ContainerStarted","Data":"a801f852557000d4bda350a1aca84f3c7968248e749a51fc2da02fa734f6e10c"} Oct 31 13:05:49.668091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.668057 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2"] Oct 31 13:05:49.673300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.673276 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.682773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.682744 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2"] Oct 31 13:05:49.784078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.783990 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbm4r\" (UniqueName: \"kubernetes.io/projected/610292dc-4e04-4d00-bd33-25a3eac7fd5f-kube-api-access-jbm4r\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.784277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.784253 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.784423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.784399 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.885568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.885518 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.885763 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.885669 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.885822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.885778 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbm4r\" (UniqueName: \"kubernetes.io/projected/610292dc-4e04-4d00-bd33-25a3eac7fd5f-kube-api-access-jbm4r\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.886515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.886483 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.888231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.888199 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.905249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.905207 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbm4r\" (UniqueName: \"kubernetes.io/projected/610292dc-4e04-4d00-bd33-25a3eac7fd5f-kube-api-access-jbm4r\") pod \"maas-api-revoke-sub-2a538cdf-gdrv2\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:49.992091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:49.992051 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:50.218060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:50.218023 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh"] Oct 31 13:05:50.313687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:50.312565 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2"] Oct 31 13:05:50.315797 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:05:50.315766 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610292dc_4e04_4d00_bd33_25a3eac7fd5f.slice/crio-2d14c195939a9b61b3fbdb92c28138b2c716b03266d7b9f93de893772016ba10 WatchSource:0}: Error finding container 2d14c195939a9b61b3fbdb92c28138b2c716b03266d7b9f93de893772016ba10: Status 404 returned error can't find the container with id 2d14c195939a9b61b3fbdb92c28138b2c716b03266d7b9f93de893772016ba10 Oct 31 13:05:50.317136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:50.317109 5074 generic.go:356] "Generic (PLEG): container finished" podID="79eeec8b-e321-471c-9e62-722bc9ba45af" containerID="9f2461493771d45a97ac8a228b00a37528ead71a4d351c67eb5d095e3dae2884" exitCode=0 Oct 31 13:05:50.317242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:50.317194 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" event={"ID":"79eeec8b-e321-471c-9e62-722bc9ba45af","Type":"ContainerDied","Data":"9f2461493771d45a97ac8a228b00a37528ead71a4d351c67eb5d095e3dae2884"} Oct 31 13:05:51.325945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.325910 5074 generic.go:356] "Generic (PLEG): container finished" podID="610292dc-4e04-4d00-bd33-25a3eac7fd5f" containerID="071395dc6c238429824d66ba4b1262fce7eaeb9778e8933a36a3221350a775fb" exitCode=0 Oct 31 13:05:51.326360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.325994 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" event={"ID":"610292dc-4e04-4d00-bd33-25a3eac7fd5f","Type":"ContainerDied","Data":"071395dc6c238429824d66ba4b1262fce7eaeb9778e8933a36a3221350a775fb"} Oct 31 13:05:51.326360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.326079 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" event={"ID":"610292dc-4e04-4d00-bd33-25a3eac7fd5f","Type":"ContainerStarted","Data":"2d14c195939a9b61b3fbdb92c28138b2c716b03266d7b9f93de893772016ba10"} Oct 31 13:05:51.572079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.572052 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:51.615835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.615803 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-openshift-ca\") pod \"79eeec8b-e321-471c-9e62-722bc9ba45af\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " Oct 31 13:05:51.616008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.615879 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-xks-ca\") pod \"79eeec8b-e321-471c-9e62-722bc9ba45af\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " Oct 31 13:05:51.616066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.616035 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79eeec8b-e321-471c-9e62-722bc9ba45af-kube-api-access-fhqf8\" (UniqueName: \"kubernetes.io/projected/79eeec8b-e321-471c-9e62-722bc9ba45af-kube-api-access-fhqf8\") pod \"79eeec8b-e321-471c-9e62-722bc9ba45af\" (UID: \"79eeec8b-e321-471c-9e62-722bc9ba45af\") " Oct 31 13:05:51.617290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.617218 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-openshift-ca" pod "79eeec8b-e321-471c-9e62-722bc9ba45af" (UID: "79eeec8b-e321-471c-9e62-722bc9ba45af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:51.619394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.619365 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-xks-ca" pod "79eeec8b-e321-471c-9e62-722bc9ba45af" (UID: "79eeec8b-e321-471c-9e62-722bc9ba45af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:51.622789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.622761 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79eeec8b-e321-471c-9e62-722bc9ba45af-kube-api-access-fhqf8" pod "79eeec8b-e321-471c-9e62-722bc9ba45af" (UID: "79eeec8b-e321-471c-9e62-722bc9ba45af"). InnerVolumeSpecName "kube-api-access-fhqf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:51.718095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.718057 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:51.718095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.718090 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79eeec8b-e321-471c-9e62-722bc9ba45af-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:51.718339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:51.718104 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqf8\" (UniqueName: \"kubernetes.io/projected/79eeec8b-e321-471c-9e62-722bc9ba45af-kube-api-access-fhqf8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:52.335426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.335393 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" Oct 31 13:05:52.335869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.335431 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9" event={"ID":"79eeec8b-e321-471c-9e62-722bc9ba45af","Type":"ContainerDied","Data":"a801f852557000d4bda350a1aca84f3c7968248e749a51fc2da02fa734f6e10c"} Oct 31 13:05:52.335869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.335477 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a801f852557000d4bda350a1aca84f3c7968248e749a51fc2da02fa734f6e10c" Oct 31 13:05:52.506594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.506555 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:52.533398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.533356 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-xks-ca\") pod \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " Oct 31 13:05:52.533594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.533428 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610292dc-4e04-4d00-bd33-25a3eac7fd5f-kube-api-access-jbm4r\" (UniqueName: \"kubernetes.io/projected/610292dc-4e04-4d00-bd33-25a3eac7fd5f-kube-api-access-jbm4r\") pod \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " Oct 31 13:05:52.533594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.533460 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-openshift-ca\") pod \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\" (UID: \"610292dc-4e04-4d00-bd33-25a3eac7fd5f\") " Oct 31 13:05:52.534215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.534181 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-openshift-ca" pod "610292dc-4e04-4d00-bd33-25a3eac7fd5f" (UID: "610292dc-4e04-4d00-bd33-25a3eac7fd5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:52.534381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.534360 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:52.538068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.538032 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-xks-ca" pod "610292dc-4e04-4d00-bd33-25a3eac7fd5f" (UID: "610292dc-4e04-4d00-bd33-25a3eac7fd5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:52.539384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.539359 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610292dc-4e04-4d00-bd33-25a3eac7fd5f-kube-api-access-jbm4r" pod "610292dc-4e04-4d00-bd33-25a3eac7fd5f" (UID: "610292dc-4e04-4d00-bd33-25a3eac7fd5f"). InnerVolumeSpecName "kube-api-access-jbm4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:52.635768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.635655 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610292dc-4e04-4d00-bd33-25a3eac7fd5f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:52.635768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:52.635688 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbm4r\" (UniqueName: \"kubernetes.io/projected/610292dc-4e04-4d00-bd33-25a3eac7fd5f-kube-api-access-jbm4r\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:53.349097 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:53.349065 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" Oct 31 13:05:53.349725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:53.349087 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2" event={"ID":"610292dc-4e04-4d00-bd33-25a3eac7fd5f","Type":"ContainerDied","Data":"2d14c195939a9b61b3fbdb92c28138b2c716b03266d7b9f93de893772016ba10"} Oct 31 13:05:53.349725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:05:53.349148 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d14c195939a9b61b3fbdb92c28138b2c716b03266d7b9f93de893772016ba10" Oct 31 13:05:55.985069 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:05:55.985031 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:06:01.300854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.300814 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts"] Oct 31 13:06:01.302476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.302450 5074 cpu_manager.go:414] "containerMap: removing container" podUID="610292dc-4e04-4d00-bd33-25a3eac7fd5f" containerName="revoke-keys" Oct 31 13:06:01.302669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.302652 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610292dc-4e04-4d00-bd33-25a3eac7fd5f" containerName="revoke-keys" Oct 31 13:06:01.302785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.302773 5074 cpu_manager.go:414] "containerMap: removing container" podUID="79eeec8b-e321-471c-9e62-722bc9ba45af" containerName="revoke-keys" Oct 31 13:06:01.302869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.302858 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79eeec8b-e321-471c-9e62-722bc9ba45af" containerName="revoke-keys" Oct 31 13:06:01.303154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.303140 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="610292dc-4e04-4d00-bd33-25a3eac7fd5f" containerName="revoke-keys" Oct 31 13:06:01.303268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.303257 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="79eeec8b-e321-471c-9e62-722bc9ba45af" containerName="revoke-keys" Oct 31 13:06:01.308617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.308596 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.324655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.319979 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:06:01.324655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.320094 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:06:01.324655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.320185 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:06:01.331355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.331330 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:06:01.380426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.380093 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts"] Oct 31 13:06:01.399806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.399763 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.420655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.400106 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.421111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.421078 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7m7\" (UniqueName: \"kubernetes.io/projected/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-kube-api-access-bj7m7\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.539531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.539499 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7m7\" (UniqueName: \"kubernetes.io/projected/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-kube-api-access-bj7m7\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.541083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.541058 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.541340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.541323 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.546171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.546144 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.549156 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.549132 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.583985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.583906 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7m7\" (UniqueName: \"kubernetes.io/projected/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-kube-api-access-bj7m7\") pod \"maas-api-revoke-sub-462e85ec-gwnts\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:01.682147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:01.682112 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:02.492445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:02.492380 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts"] Oct 31 13:06:02.548850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:02.548806 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" event={"ID":"1e62ae1b-f87e-465d-aeaa-3b8785a28e03","Type":"ContainerStarted","Data":"13c1a3402dd7ec0e246aa5bf4ddb3e4d9bb5853fbdda40588a317bc3f3f73968"} Oct 31 13:06:03.384655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:03.384568 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:03.389656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:03.389349 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:03.634388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:03.634345 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" event={"ID":"1e62ae1b-f87e-465d-aeaa-3b8785a28e03","Type":"ContainerStarted","Data":"697c283022db83f1f4e562546467f145409c4682510cd7a8d350633f98f9b889"} Oct 31 13:06:03.680405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:03.680295 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" podStartSLOduration=2.680276161 podStartE2EDuration="2.680276161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:03.679770559 +0000 UTC m=+2057.857488544" watchObservedRunningTime="2026-10-31 13:06:03.680276161 +0000 UTC m=+2057.857994144" Oct 31 13:06:04.669015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:04.668976 5074 generic.go:356] "Generic (PLEG): container finished" podID="1e62ae1b-f87e-465d-aeaa-3b8785a28e03" containerID="697c283022db83f1f4e562546467f145409c4682510cd7a8d350633f98f9b889" exitCode=0 Oct 31 13:06:04.669848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:04.669812 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" event={"ID":"1e62ae1b-f87e-465d-aeaa-3b8785a28e03","Type":"ContainerDied","Data":"697c283022db83f1f4e562546467f145409c4682510cd7a8d350633f98f9b889"} Oct 31 13:06:05.241182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.240709 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" podUID="a4e854b7-4ecb-45bd-a27c-b66143b75506" containerName="main" containerID="cri-o://f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832" gracePeriod=45 Oct 31 13:06:05.546702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.546674 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:06:05.641173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641135 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e854b7-4ecb-45bd-a27c-b66143b75506-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e854b7-4ecb-45bd-a27c-b66143b75506-tls-certs\") pod \"a4e854b7-4ecb-45bd-a27c-b66143b75506\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " Oct 31 13:06:05.641329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641218 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-model-cache\") pod \"a4e854b7-4ecb-45bd-a27c-b66143b75506\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " Oct 31 13:06:05.641329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641252 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-dshm\") pod \"a4e854b7-4ecb-45bd-a27c-b66143b75506\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " Oct 31 13:06:05.641329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641283 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-home\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-home\") pod \"a4e854b7-4ecb-45bd-a27c-b66143b75506\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " Oct 31 13:06:05.641436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641331 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-tmp-dir\") pod \"a4e854b7-4ecb-45bd-a27c-b66143b75506\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " Oct 31 13:06:05.641436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641379 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e854b7-4ecb-45bd-a27c-b66143b75506-kube-api-access-rmstj\" (UniqueName: \"kubernetes.io/projected/a4e854b7-4ecb-45bd-a27c-b66143b75506-kube-api-access-rmstj\") pod \"a4e854b7-4ecb-45bd-a27c-b66143b75506\" (UID: \"a4e854b7-4ecb-45bd-a27c-b66143b75506\") " Oct 31 13:06:05.641601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641515 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-model-cache" pod "a4e854b7-4ecb-45bd-a27c-b66143b75506" (UID: "a4e854b7-4ecb-45bd-a27c-b66143b75506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:05.641601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641569 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-home" pod "a4e854b7-4ecb-45bd-a27c-b66143b75506" (UID: "a4e854b7-4ecb-45bd-a27c-b66143b75506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:05.641754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641624 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-tmp-dir" pod "a4e854b7-4ecb-45bd-a27c-b66143b75506" (UID: "a4e854b7-4ecb-45bd-a27c-b66143b75506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:05.641894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641876 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:05.641992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641895 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:05.641992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.641908 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:05.643521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.643492 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e854b7-4ecb-45bd-a27c-b66143b75506-tls-certs" pod "a4e854b7-4ecb-45bd-a27c-b66143b75506" (UID: "a4e854b7-4ecb-45bd-a27c-b66143b75506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:05.643686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.643606 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e854b7-4ecb-45bd-a27c-b66143b75506-kube-api-access-rmstj" pod "a4e854b7-4ecb-45bd-a27c-b66143b75506" (UID: "a4e854b7-4ecb-45bd-a27c-b66143b75506"). InnerVolumeSpecName "kube-api-access-rmstj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:05.643863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.643841 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-dshm" pod "a4e854b7-4ecb-45bd-a27c-b66143b75506" (UID: "a4e854b7-4ecb-45bd-a27c-b66143b75506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:05.680299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.680266 5074 generic.go:356] "Generic (PLEG): container finished" podID="a4e854b7-4ecb-45bd-a27c-b66143b75506" containerID="f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832" exitCode=0 Oct 31 13:06:05.680732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.680315 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" Oct 31 13:06:05.680732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.680346 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" event={"ID":"a4e854b7-4ecb-45bd-a27c-b66143b75506","Type":"ContainerDied","Data":"f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832"} Oct 31 13:06:05.680732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.680390 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh" event={"ID":"a4e854b7-4ecb-45bd-a27c-b66143b75506","Type":"ContainerDied","Data":"94bdef28abeb07cfbb0c3c24d2ad128368264de460bb716892b258c18c6e650c"} Oct 31 13:06:05.680732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.680409 5074 scope.go:122] "RemoveContainer" containerID="f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832" Oct 31 13:06:05.707424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.707385 5074 scope.go:122] "RemoveContainer" containerID="f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832" Oct 31 13:06:05.712699 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:06:05.710802 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832\": container with ID starting with f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832 not found: ID does not exist" containerID="f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832" Oct 31 13:06:05.712699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.710852 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832"} err="failed to get container status \"f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832\": rpc error: code = NotFound desc = could not find container \"f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832\": container with ID starting with f3140f6ded505ab72c6c78c7ec422c4711b1c95b31c6bf7c289e6608e709d832 not found: ID does not exist" Oct 31 13:06:05.724565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.724532 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh"] Oct 31 13:06:05.728695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.728553 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4989f9/ipp-route-4989f9-kserve-5dfd84797c-ph6lh"] Oct 31 13:06:05.742972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.742946 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e854b7-4ecb-45bd-a27c-b66143b75506-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:05.743132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.743122 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e854b7-4ecb-45bd-a27c-b66143b75506-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:05.743245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.743222 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmstj\" (UniqueName: \"kubernetes.io/projected/a4e854b7-4ecb-45bd-a27c-b66143b75506-kube-api-access-rmstj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:05.848927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.848905 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:05.942107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.941527 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e854b7-4ecb-45bd-a27c-b66143b75506" path="/var/lib/kubelet/pods/a4e854b7-4ecb-45bd-a27c-b66143b75506/volumes" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.944558 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl"] Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.944737 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-kube-api-access-bj7m7\" (UniqueName: \"kubernetes.io/projected/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-kube-api-access-bj7m7\") pod \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.944878 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-openshift-ca\") pod \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945023 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-xks-ca\") pod \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\" (UID: \"1e62ae1b-f87e-465d-aeaa-3b8785a28e03\") " Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945248 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-openshift-ca" pod "1e62ae1b-f87e-465d-aeaa-3b8785a28e03" (UID: "1e62ae1b-f87e-465d-aeaa-3b8785a28e03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945379 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a4e854b7-4ecb-45bd-a27c-b66143b75506" containerName="main" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945397 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e854b7-4ecb-45bd-a27c-b66143b75506" containerName="main" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945470 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1e62ae1b-f87e-465d-aeaa-3b8785a28e03" containerName="revoke-keys" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945479 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e62ae1b-f87e-465d-aeaa-3b8785a28e03" containerName="revoke-keys" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945647 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e854b7-4ecb-45bd-a27c-b66143b75506" containerName="main" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945672 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e62ae1b-f87e-465d-aeaa-3b8785a28e03" containerName="revoke-keys" Oct 31 13:06:05.947125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.945728 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:05.947685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.947369 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-xks-ca" pod "1e62ae1b-f87e-465d-aeaa-3b8785a28e03" (UID: "1e62ae1b-f87e-465d-aeaa-3b8785a28e03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:05.947685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.947670 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-kube-api-access-bj7m7" pod "1e62ae1b-f87e-465d-aeaa-3b8785a28e03" (UID: "1e62ae1b-f87e-465d-aeaa-3b8785a28e03"). InnerVolumeSpecName "kube-api-access-bj7m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:05.949467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.949445 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl"] Oct 31 13:06:05.949589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:05.949546 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.047257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.047216 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxb4\" (UniqueName: \"kubernetes.io/projected/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-kube-api-access-zlxb4\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.047426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.047274 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.047426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.047354 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.047585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.047567 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:06.047620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.047589 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7m7\" (UniqueName: \"kubernetes.io/projected/1e62ae1b-f87e-465d-aeaa-3b8785a28e03-kube-api-access-bj7m7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:06.154583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.153975 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxb4\" (UniqueName: \"kubernetes.io/projected/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-kube-api-access-zlxb4\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.154583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.154555 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.154935 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.154599 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.155901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.155718 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.165551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.165514 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.167958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.167934 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxb4\" (UniqueName: \"kubernetes.io/projected/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-kube-api-access-zlxb4\") pod \"maas-api-revoke-sub-c873e904-f42bl\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.266410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.266381 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:06.555961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.555929 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl"] Oct 31 13:06:06.556143 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:06:06.556116 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fada12e_b9a1_4f24_9e7b_8b8a3a41e548.slice/crio-69db290dfe20cecb86e35727aa1b07e3b8ec0dad5871d1e14d8843fefc523f52 WatchSource:0}: Error finding container 69db290dfe20cecb86e35727aa1b07e3b8ec0dad5871d1e14d8843fefc523f52: Status 404 returned error can't find the container with id 69db290dfe20cecb86e35727aa1b07e3b8ec0dad5871d1e14d8843fefc523f52 Oct 31 13:06:06.689846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.689815 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" Oct 31 13:06:06.689846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.689831 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts" event={"ID":"1e62ae1b-f87e-465d-aeaa-3b8785a28e03","Type":"ContainerDied","Data":"13c1a3402dd7ec0e246aa5bf4ddb3e4d9bb5853fbdda40588a317bc3f3f73968"} Oct 31 13:06:06.690340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.689888 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c1a3402dd7ec0e246aa5bf4ddb3e4d9bb5853fbdda40588a317bc3f3f73968" Oct 31 13:06:06.692490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:06.692461 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" event={"ID":"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548","Type":"ContainerStarted","Data":"69db290dfe20cecb86e35727aa1b07e3b8ec0dad5871d1e14d8843fefc523f52"} Oct 31 13:06:07.707505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:07.707469 5074 generic.go:356] "Generic (PLEG): container finished" podID="0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" containerID="ceb20ecd49ce98013f4864867a48509b8def8af4a94712f82d3c76e8e6c193f4" exitCode=0 Oct 31 13:06:07.707912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:07.707547 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" event={"ID":"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548","Type":"ContainerDied","Data":"ceb20ecd49ce98013f4864867a48509b8def8af4a94712f82d3c76e8e6c193f4"} Oct 31 13:06:08.916628 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:08.916292 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:09.006892 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.006797 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-xks-ca\") pod \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " Oct 31 13:06:09.006892 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.006870 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-kube-api-access-zlxb4\" (UniqueName: \"kubernetes.io/projected/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-kube-api-access-zlxb4\") pod \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " Oct 31 13:06:09.007121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.007022 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-openshift-ca\") pod \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\" (UID: \"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548\") " Oct 31 13:06:09.007375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.007349 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-openshift-ca" pod "0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" (UID: "0fada12e-b9a1-4f24-9e7b-8b8a3a41e548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:09.007523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.007504 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:09.009139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.009116 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-kube-api-access-zlxb4" pod "0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" (UID: "0fada12e-b9a1-4f24-9e7b-8b8a3a41e548"). InnerVolumeSpecName "kube-api-access-zlxb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:09.009388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.009370 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-xks-ca" pod "0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" (UID: "0fada12e-b9a1-4f24-9e7b-8b8a3a41e548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:09.108716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.108676 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:09.108716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.108714 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxb4\" (UniqueName: \"kubernetes.io/projected/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548-kube-api-access-zlxb4\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:09.728874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.728685 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" event={"ID":"0fada12e-b9a1-4f24-9e7b-8b8a3a41e548","Type":"ContainerDied","Data":"69db290dfe20cecb86e35727aa1b07e3b8ec0dad5871d1e14d8843fefc523f52"} Oct 31 13:06:09.729172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.729138 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69db290dfe20cecb86e35727aa1b07e3b8ec0dad5871d1e14d8843fefc523f52" Oct 31 13:06:09.729333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:09.728912 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl" Oct 31 13:06:14.268452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.268418 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5"] Oct 31 13:06:14.269221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.269199 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" containerName="revoke-keys" Oct 31 13:06:14.269365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.269223 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" containerName="revoke-keys" Oct 31 13:06:14.269473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.269458 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" containerName="revoke-keys" Oct 31 13:06:14.273278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.273255 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.284268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.284241 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:06:14.284373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.284319 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:06:14.285648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.285615 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:06:14.285746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.285717 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:06:14.286940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.286917 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5"] Oct 31 13:06:14.387837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.387804 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24573578-5955-4993-a96b-dc181b2a237e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.388070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.388047 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24573578-5955-4993-a96b-dc181b2a237e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.388146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.388088 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zr2s\" (UniqueName: \"kubernetes.io/projected/24573578-5955-4993-a96b-dc181b2a237e-kube-api-access-6zr2s\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.489547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.489514 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24573578-5955-4993-a96b-dc181b2a237e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.489547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.489555 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zr2s\" (UniqueName: \"kubernetes.io/projected/24573578-5955-4993-a96b-dc181b2a237e-kube-api-access-6zr2s\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.489786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.489617 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24573578-5955-4993-a96b-dc181b2a237e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.490374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.490350 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24573578-5955-4993-a96b-dc181b2a237e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.491955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.491932 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24573578-5955-4993-a96b-dc181b2a237e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.501998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.501972 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zr2s\" (UniqueName: \"kubernetes.io/projected/24573578-5955-4993-a96b-dc181b2a237e-kube-api-access-6zr2s\") pod \"maas-api-revoke-sub-82fed814-4z5b5\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.608055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.607974 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:14.909913 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:06:14.909876 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24573578_5955_4993_a96b_dc181b2a237e.slice/crio-4806ea315e20d5fc8c9d2ca082b8eea625d4af0cea80d30180666ab69714803b WatchSource:0}: Error finding container 4806ea315e20d5fc8c9d2ca082b8eea625d4af0cea80d30180666ab69714803b: Status 404 returned error can't find the container with id 4806ea315e20d5fc8c9d2ca082b8eea625d4af0cea80d30180666ab69714803b Oct 31 13:06:14.911513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:14.911483 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5"] Oct 31 13:06:15.799866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:15.799745 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" event={"ID":"24573578-5955-4993-a96b-dc181b2a237e","Type":"ContainerStarted","Data":"8f417946c781b3312881959576a6675c9a62dd93ed9e908811125765f8117e3b"} Oct 31 13:06:15.799866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:15.799815 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" event={"ID":"24573578-5955-4993-a96b-dc181b2a237e","Type":"ContainerStarted","Data":"4806ea315e20d5fc8c9d2ca082b8eea625d4af0cea80d30180666ab69714803b"} Oct 31 13:06:15.822944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:15.822888 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" podStartSLOduration=1.822868541 podStartE2EDuration="1.822868541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:15.82033283 +0000 UTC m=+2069.998050825" watchObservedRunningTime="2026-10-31 13:06:15.822868541 +0000 UTC m=+2070.000586526" Oct 31 13:06:16.811921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:16.811886 5074 generic.go:356] "Generic (PLEG): container finished" podID="24573578-5955-4993-a96b-dc181b2a237e" containerID="8f417946c781b3312881959576a6675c9a62dd93ed9e908811125765f8117e3b" exitCode=0 Oct 31 13:06:16.812334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:16.811973 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" event={"ID":"24573578-5955-4993-a96b-dc181b2a237e","Type":"ContainerDied","Data":"8f417946c781b3312881959576a6675c9a62dd93ed9e908811125765f8117e3b"} Oct 31 13:06:17.992979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:17.992948 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:18.043115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.043076 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24573578-5955-4993-a96b-dc181b2a237e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24573578-5955-4993-a96b-dc181b2a237e-maas-api-openshift-ca\") pod \"24573578-5955-4993-a96b-dc181b2a237e\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " Oct 31 13:06:18.043283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.043167 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24573578-5955-4993-a96b-dc181b2a237e-kube-api-access-6zr2s\" (UniqueName: \"kubernetes.io/projected/24573578-5955-4993-a96b-dc181b2a237e-kube-api-access-6zr2s\") pod \"24573578-5955-4993-a96b-dc181b2a237e\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " Oct 31 13:06:18.043416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.043388 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24573578-5955-4993-a96b-dc181b2a237e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24573578-5955-4993-a96b-dc181b2a237e-maas-api-xks-ca\") pod \"24573578-5955-4993-a96b-dc181b2a237e\" (UID: \"24573578-5955-4993-a96b-dc181b2a237e\") " Oct 31 13:06:18.043550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.043515 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24573578-5955-4993-a96b-dc181b2a237e-maas-api-openshift-ca" pod "24573578-5955-4993-a96b-dc181b2a237e" (UID: "24573578-5955-4993-a96b-dc181b2a237e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:18.044071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.044046 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24573578-5955-4993-a96b-dc181b2a237e-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:18.045485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.045457 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24573578-5955-4993-a96b-dc181b2a237e-maas-api-xks-ca" pod "24573578-5955-4993-a96b-dc181b2a237e" (UID: "24573578-5955-4993-a96b-dc181b2a237e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:18.045918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.045894 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24573578-5955-4993-a96b-dc181b2a237e-kube-api-access-6zr2s" pod "24573578-5955-4993-a96b-dc181b2a237e" (UID: "24573578-5955-4993-a96b-dc181b2a237e"). InnerVolumeSpecName "kube-api-access-6zr2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:18.145540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.145464 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24573578-5955-4993-a96b-dc181b2a237e-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:18.145540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.145495 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zr2s\" (UniqueName: \"kubernetes.io/projected/24573578-5955-4993-a96b-dc181b2a237e-kube-api-access-6zr2s\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:18.828397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.828364 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" Oct 31 13:06:18.828397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.828377 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5" event={"ID":"24573578-5955-4993-a96b-dc181b2a237e","Type":"ContainerDied","Data":"4806ea315e20d5fc8c9d2ca082b8eea625d4af0cea80d30180666ab69714803b"} Oct 31 13:06:18.828740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:18.828476 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4806ea315e20d5fc8c9d2ca082b8eea625d4af0cea80d30180666ab69714803b" Oct 31 13:06:23.374549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:23.374518 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:06:26.151066 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:06:26.151020 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:06:26.373912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:26.373866 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 31 13:06:27.090768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.090720 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.220260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.197176 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.223844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.221726 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.229584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.228175 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.229584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.228528 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 31 13:06:27.273212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.273171 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 13:06:27.278761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.274167 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.383878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.383773 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.386343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.384863 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:06:27.404998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.404013 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.419112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.419068 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:27.693285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.693205 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:06:27.841818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:27.841772 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:28.264734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.264700 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h"] Oct 31 13:06:28.265394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.265376 5074 cpu_manager.go:414] "containerMap: removing container" podUID="24573578-5955-4993-a96b-dc181b2a237e" containerName="revoke-keys" Oct 31 13:06:28.265448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.265397 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24573578-5955-4993-a96b-dc181b2a237e" containerName="revoke-keys" Oct 31 13:06:28.265522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.265512 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24573578-5955-4993-a96b-dc181b2a237e" containerName="revoke-keys" Oct 31 13:06:28.270382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.270356 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.301839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.301809 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:06:28.302901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.302873 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:06:28.303058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.302951 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:06:28.303058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.303043 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:06:28.307984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.307953 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h"] Oct 31 13:06:28.401158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.401121 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.401340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.401225 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.401436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.401382 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f29qf\" (UniqueName: \"kubernetes.io/projected/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-kube-api-access-f29qf\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.504755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.504714 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.504942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.504804 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.504942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.504889 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f29qf\" (UniqueName: \"kubernetes.io/projected/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-kube-api-access-f29qf\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.505928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.505848 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.508209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.508185 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.517372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.517315 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f29qf\" (UniqueName: \"kubernetes.io/projected/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-kube-api-access-f29qf\") pod \"maas-api-revoke-sub-bebdef22-fzj2h\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.617143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.617107 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:28.901742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:28.901711 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h"] Oct 31 13:06:28.904834 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:06:28.904794 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2d8c11_b693_4863_8e0d_fb6daac29b0c.slice/crio-79e8bc1b3d6ccec8f30660e3c00acd0ad3d97aa07898a758f11f96ee721a3a74 WatchSource:0}: Error finding container 79e8bc1b3d6ccec8f30660e3c00acd0ad3d97aa07898a758f11f96ee721a3a74: Status 404 returned error can't find the container with id 79e8bc1b3d6ccec8f30660e3c00acd0ad3d97aa07898a758f11f96ee721a3a74 Oct 31 13:06:29.354326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:29.354284 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" event={"ID":"bd2d8c11-b693-4863-8e0d-fb6daac29b0c","Type":"ContainerStarted","Data":"968226a9e28e620d3dfa78d2637000790061d56c54a927c0c2d5745bc9fe36cb"} Oct 31 13:06:29.354800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:29.354346 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" event={"ID":"bd2d8c11-b693-4863-8e0d-fb6daac29b0c","Type":"ContainerStarted","Data":"79e8bc1b3d6ccec8f30660e3c00acd0ad3d97aa07898a758f11f96ee721a3a74"} Oct 31 13:06:29.379554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:29.379501 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" podStartSLOduration=1.379487329 podStartE2EDuration="1.379487329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:29.372177424 +0000 UTC m=+2083.549895407" watchObservedRunningTime="2026-10-31 13:06:29.379487329 +0000 UTC m=+2083.557205311" Oct 31 13:06:29.906348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:29.906309 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf"] Oct 31 13:06:29.911579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:29.911554 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:29.920403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:29.920374 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf"] Oct 31 13:06:30.030849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.030813 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.031026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.031003 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.031089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.031068 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnt5g\" (UniqueName: \"kubernetes.io/projected/05e3eea9-248a-4f97-add6-8dc68e27d191-kube-api-access-tnt5g\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.132955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.132868 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.133126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.132959 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt5g\" (UniqueName: \"kubernetes.io/projected/05e3eea9-248a-4f97-add6-8dc68e27d191-kube-api-access-tnt5g\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.133126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.133115 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.133575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.133555 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.135528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.135503 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.145772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.145741 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnt5g\" (UniqueName: \"kubernetes.io/projected/05e3eea9-248a-4f97-add6-8dc68e27d191-kube-api-access-tnt5g\") pod \"maas-api-revoke-sub-fe481ae2-99tkf\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.234440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.234407 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:30.382660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.381193 5074 generic.go:356] "Generic (PLEG): container finished" podID="bd2d8c11-b693-4863-8e0d-fb6daac29b0c" containerID="968226a9e28e620d3dfa78d2637000790061d56c54a927c0c2d5745bc9fe36cb" exitCode=0 Oct 31 13:06:30.392585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.391597 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" event={"ID":"bd2d8c11-b693-4863-8e0d-fb6daac29b0c","Type":"ContainerDied","Data":"968226a9e28e620d3dfa78d2637000790061d56c54a927c0c2d5745bc9fe36cb"} Oct 31 13:06:30.523448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:30.523417 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf"] Oct 31 13:06:30.523894 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:06:30.523872 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e3eea9_248a_4f97_add6_8dc68e27d191.slice/crio-a829575d9fe52d8d04239ad1cd5db99ed20d0b6d7e441893d5c3fde233fd86b1 WatchSource:0}: Error finding container a829575d9fe52d8d04239ad1cd5db99ed20d0b6d7e441893d5c3fde233fd86b1: Status 404 returned error can't find the container with id a829575d9fe52d8d04239ad1cd5db99ed20d0b6d7e441893d5c3fde233fd86b1 Oct 31 13:06:31.404239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.404203 5074 generic.go:356] "Generic (PLEG): container finished" podID="05e3eea9-248a-4f97-add6-8dc68e27d191" containerID="59c655c487c9cb271cd4260b3b9b4828a4aa3d13d401b7e13fc8a4af7766bb9d" exitCode=0 Oct 31 13:06:31.404751 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.404723 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" event={"ID":"05e3eea9-248a-4f97-add6-8dc68e27d191","Type":"ContainerDied","Data":"59c655c487c9cb271cd4260b3b9b4828a4aa3d13d401b7e13fc8a4af7766bb9d"} Oct 31 13:06:31.404823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.404785 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" event={"ID":"05e3eea9-248a-4f97-add6-8dc68e27d191","Type":"ContainerStarted","Data":"a829575d9fe52d8d04239ad1cd5db99ed20d0b6d7e441893d5c3fde233fd86b1"} Oct 31 13:06:31.621372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.621305 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:31.778983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.778948 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-openshift-ca\") pod \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " Oct 31 13:06:31.779185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.779083 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-kube-api-access-f29qf\" (UniqueName: \"kubernetes.io/projected/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-kube-api-access-f29qf\") pod \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " Oct 31 13:06:31.779185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.779138 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-xks-ca\") pod \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\" (UID: \"bd2d8c11-b693-4863-8e0d-fb6daac29b0c\") " Oct 31 13:06:31.779360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.779335 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-openshift-ca" pod "bd2d8c11-b693-4863-8e0d-fb6daac29b0c" (UID: "bd2d8c11-b693-4863-8e0d-fb6daac29b0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:31.779855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.779828 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:31.782142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.782108 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-xks-ca" pod "bd2d8c11-b693-4863-8e0d-fb6daac29b0c" (UID: "bd2d8c11-b693-4863-8e0d-fb6daac29b0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:31.782142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.782121 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-kube-api-access-f29qf" pod "bd2d8c11-b693-4863-8e0d-fb6daac29b0c" (UID: "bd2d8c11-b693-4863-8e0d-fb6daac29b0c"). InnerVolumeSpecName "kube-api-access-f29qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:31.882353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.882251 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f29qf\" (UniqueName: \"kubernetes.io/projected/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-kube-api-access-f29qf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:31.882353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:31.882306 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2d8c11-b693-4863-8e0d-fb6daac29b0c-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:32.418871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.418837 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" Oct 31 13:06:32.419427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.418881 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h" event={"ID":"bd2d8c11-b693-4863-8e0d-fb6daac29b0c","Type":"ContainerDied","Data":"79e8bc1b3d6ccec8f30660e3c00acd0ad3d97aa07898a758f11f96ee721a3a74"} Oct 31 13:06:32.419427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.418970 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e8bc1b3d6ccec8f30660e3c00acd0ad3d97aa07898a758f11f96ee721a3a74" Oct 31 13:06:32.651025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.650988 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:32.717476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.711712 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e3eea9-248a-4f97-add6-8dc68e27d191-kube-api-access-tnt5g\" (UniqueName: \"kubernetes.io/projected/05e3eea9-248a-4f97-add6-8dc68e27d191-kube-api-access-tnt5g\") pod \"05e3eea9-248a-4f97-add6-8dc68e27d191\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " Oct 31 13:06:32.717476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.711804 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-openshift-ca\") pod \"05e3eea9-248a-4f97-add6-8dc68e27d191\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " Oct 31 13:06:32.717476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.711871 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-xks-ca\") pod \"05e3eea9-248a-4f97-add6-8dc68e27d191\" (UID: \"05e3eea9-248a-4f97-add6-8dc68e27d191\") " Oct 31 13:06:32.717476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.713003 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-openshift-ca" pod "05e3eea9-248a-4f97-add6-8dc68e27d191" (UID: "05e3eea9-248a-4f97-add6-8dc68e27d191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:32.733689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.733623 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-xks-ca" pod "05e3eea9-248a-4f97-add6-8dc68e27d191" (UID: "05e3eea9-248a-4f97-add6-8dc68e27d191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:32.741706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.741672 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e3eea9-248a-4f97-add6-8dc68e27d191-kube-api-access-tnt5g" pod "05e3eea9-248a-4f97-add6-8dc68e27d191" (UID: "05e3eea9-248a-4f97-add6-8dc68e27d191"). InnerVolumeSpecName "kube-api-access-tnt5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:32.842655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.834693 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnt5g\" (UniqueName: \"kubernetes.io/projected/05e3eea9-248a-4f97-add6-8dc68e27d191-kube-api-access-tnt5g\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:32.842655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.834731 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:32.842655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:32.834750 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05e3eea9-248a-4f97-add6-8dc68e27d191-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:33.432798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:33.432761 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" event={"ID":"05e3eea9-248a-4f97-add6-8dc68e27d191","Type":"ContainerDied","Data":"a829575d9fe52d8d04239ad1cd5db99ed20d0b6d7e441893d5c3fde233fd86b1"} Oct 31 13:06:33.433365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:33.432816 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a829575d9fe52d8d04239ad1cd5db99ed20d0b6d7e441893d5c3fde233fd86b1" Oct 31 13:06:33.433365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:33.432871 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf" Oct 31 13:06:46.311704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.311676 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:06:46.312369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.312348 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:06:46.335314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.335283 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982"] Oct 31 13:06:46.335905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.335888 5074 cpu_manager.go:414] "containerMap: removing container" podUID="bd2d8c11-b693-4863-8e0d-fb6daac29b0c" containerName="revoke-keys" Oct 31 13:06:46.335978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.335906 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2d8c11-b693-4863-8e0d-fb6daac29b0c" containerName="revoke-keys" Oct 31 13:06:46.335978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.335960 5074 cpu_manager.go:414] "containerMap: removing container" podUID="05e3eea9-248a-4f97-add6-8dc68e27d191" containerName="revoke-keys" Oct 31 13:06:46.335978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.335965 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e3eea9-248a-4f97-add6-8dc68e27d191" containerName="revoke-keys" Oct 31 13:06:46.336107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.336094 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e3eea9-248a-4f97-add6-8dc68e27d191" containerName="revoke-keys" Oct 31 13:06:46.336141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.336109 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2d8c11-b693-4863-8e0d-fb6daac29b0c" containerName="revoke-keys" Oct 31 13:06:46.339766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.339747 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.344679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.344535 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:06:46.347594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.347359 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:06:46.347724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.347684 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:06:46.347846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.347725 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:06:46.350819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.350796 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982"] Oct 31 13:06:46.386827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.386793 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.386993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.386850 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrhq\" (UniqueName: \"kubernetes.io/projected/44fdcfa4-e254-4672-b2ed-8b052fe65de2-kube-api-access-zdrhq\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.386993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.386880 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.489692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.489495 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.497861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.490753 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.498112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.497742 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrhq\" (UniqueName: \"kubernetes.io/projected/44fdcfa4-e254-4672-b2ed-8b052fe65de2-kube-api-access-zdrhq\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.498292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.498272 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.507009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.506963 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.517690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.516248 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrhq\" (UniqueName: \"kubernetes.io/projected/44fdcfa4-e254-4672-b2ed-8b052fe65de2-kube-api-access-zdrhq\") pod \"maas-api-revoke-sub-2c3f14dc-lr982\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:46.663240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:46.663148 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:47.249658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:47.240861 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982"] Oct 31 13:06:47.766234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:47.766190 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" event={"ID":"44fdcfa4-e254-4672-b2ed-8b052fe65de2","Type":"ContainerStarted","Data":"989c5678b6e2da4e2a2656f4b3eee02200072256e7e9fddd3d850ba0685b5dd5"} Oct 31 13:06:48.500656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:48.499843 5074 scope.go:122] "RemoveContainer" containerID="c3bc704885c38521fbb50aa34bb6f38d0b8d67215a82a3fd177cf19e454542cf" Oct 31 13:06:48.806742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:48.749251 5074 scope.go:122] "RemoveContainer" containerID="9423210e6000d0eb2ac773ac6e34f88141717f1f6022c6a23815cf1f461aa800" Oct 31 13:06:48.902671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:48.898709 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:48.971423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:48.971374 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.055686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.052170 5074 scope.go:122] "RemoveContainer" containerID="72787591a08e59ab4ea43c0aa69288013358771d0e5c600b91f795eb0c1e8e8a" Oct 31 13:06:49.139657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.108192 5074 generic.go:356] "Generic (PLEG): container finished" podID="44fdcfa4-e254-4672-b2ed-8b052fe65de2" containerID="5ffdf902cac1aa033a2aadd7950c58c357195fb299f96a5812d95b8c468a6f71" exitCode=0 Oct 31 13:06:49.139657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.108308 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" event={"ID":"44fdcfa4-e254-4672-b2ed-8b052fe65de2","Type":"ContainerDied","Data":"5ffdf902cac1aa033a2aadd7950c58c357195fb299f96a5812d95b8c468a6f71"} Oct 31 13:06:49.204656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.204452 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.227836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.216910 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.393862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.393801 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.394995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.394926 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.409802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.409735 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 13:06:49.535657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.528697 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4"] Oct 31 13:06:49.592801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.592749 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.635541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.635451 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4"] Oct 31 13:06:49.693095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.693052 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.693451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.693428 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.698853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.698826 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7mz\" (UniqueName: \"kubernetes.io/projected/1d3affa8-4f80-428f-ac43-8747dde6ba0f-kube-api-access-rc7mz\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.783105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.761053 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.804839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.804752 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.805138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.805114 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.805380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.805361 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7mz\" (UniqueName: \"kubernetes.io/projected/1d3affa8-4f80-428f-ac43-8747dde6ba0f-kube-api-access-rc7mz\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.809103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.809084 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.815628 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.815569 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.845507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.844999 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:06:49.867026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.865985 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7mz\" (UniqueName: \"kubernetes.io/projected/1d3affa8-4f80-428f-ac43-8747dde6ba0f-kube-api-access-rc7mz\") pod \"maas-api-revoke-sub-264bc5cd-gqlt4\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:49.949289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:49.918202 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Oct 31 13:06:50.160922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.160890 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:50.319380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.319346 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:50.325175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.325143 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-xks-ca\") pod \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " Oct 31 13:06:50.325384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.325307 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fdcfa4-e254-4672-b2ed-8b052fe65de2-kube-api-access-zdrhq\" (UniqueName: \"kubernetes.io/projected/44fdcfa4-e254-4672-b2ed-8b052fe65de2-kube-api-access-zdrhq\") pod \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " Oct 31 13:06:50.325529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.325513 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-openshift-ca\") pod \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\" (UID: \"44fdcfa4-e254-4672-b2ed-8b052fe65de2\") " Oct 31 13:06:50.325940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.325907 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-openshift-ca" pod "44fdcfa4-e254-4672-b2ed-8b052fe65de2" (UID: "44fdcfa4-e254-4672-b2ed-8b052fe65de2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:50.326125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.326104 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:50.327561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.327535 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-xks-ca" pod "44fdcfa4-e254-4672-b2ed-8b052fe65de2" (UID: "44fdcfa4-e254-4672-b2ed-8b052fe65de2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:50.328907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.328885 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fdcfa4-e254-4672-b2ed-8b052fe65de2-kube-api-access-zdrhq" pod "44fdcfa4-e254-4672-b2ed-8b052fe65de2" (UID: "44fdcfa4-e254-4672-b2ed-8b052fe65de2"). InnerVolumeSpecName "kube-api-access-zdrhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:50.427496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.427461 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrhq\" (UniqueName: \"kubernetes.io/projected/44fdcfa4-e254-4672-b2ed-8b052fe65de2-kube-api-access-zdrhq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:50.427496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.427495 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fdcfa4-e254-4672-b2ed-8b052fe65de2-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:50.513610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:50.509917 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4"] Oct 31 13:06:51.126865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.126779 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" Oct 31 13:06:51.126865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.126791 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982" event={"ID":"44fdcfa4-e254-4672-b2ed-8b052fe65de2","Type":"ContainerDied","Data":"989c5678b6e2da4e2a2656f4b3eee02200072256e7e9fddd3d850ba0685b5dd5"} Oct 31 13:06:51.126865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.126848 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="989c5678b6e2da4e2a2656f4b3eee02200072256e7e9fddd3d850ba0685b5dd5" Oct 31 13:06:51.129137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.129113 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" event={"ID":"1d3affa8-4f80-428f-ac43-8747dde6ba0f","Type":"ContainerStarted","Data":"82b751d33530e2be1cbf55ffed130e665257e2331de25c08a6eda31da096886b"} Oct 31 13:06:51.129278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.129160 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" event={"ID":"1d3affa8-4f80-428f-ac43-8747dde6ba0f","Type":"ContainerStarted","Data":"2328aa652c79e39a759ea6f3b6722fe325b5f787dbec6dbe97fd2c9721a5927f"} Oct 31 13:06:51.154925 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.154868 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" podStartSLOduration=2.154851496 podStartE2EDuration="2.154851496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:51.151126118 +0000 UTC m=+2105.328844103" watchObservedRunningTime="2026-10-31 13:06:51.154851496 +0000 UTC m=+2105.332569480" Oct 31 13:06:51.433390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.433284 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7"] Oct 31 13:06:51.433660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:51.433609 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" containerID="cri-o://3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc" gracePeriod=30 Oct 31 13:06:52.142498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:52.142465 5074 generic.go:356] "Generic (PLEG): container finished" podID="1d3affa8-4f80-428f-ac43-8747dde6ba0f" containerID="82b751d33530e2be1cbf55ffed130e665257e2331de25c08a6eda31da096886b" exitCode=0 Oct 31 13:06:52.143052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:52.142553 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" event={"ID":"1d3affa8-4f80-428f-ac43-8747dde6ba0f","Type":"ContainerDied","Data":"82b751d33530e2be1cbf55ffed130e665257e2331de25c08a6eda31da096886b"} Oct 31 13:06:53.230475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.230426 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w"] Oct 31 13:06:53.232273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.232240 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" containerID="cri-o://39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2" gracePeriod=30 Oct 31 13:06:53.249738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.249452 5074 patch_prober.go:29] interesting pod/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:06:53.249738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.249507 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:06:53.348111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.348082 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:53.460749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.460658 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-xks-ca\") pod \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " Oct 31 13:06:53.460749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.460745 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3affa8-4f80-428f-ac43-8747dde6ba0f-kube-api-access-rc7mz\" (UniqueName: \"kubernetes.io/projected/1d3affa8-4f80-428f-ac43-8747dde6ba0f-kube-api-access-rc7mz\") pod \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " Oct 31 13:06:53.460991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.460949 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-openshift-ca\") pod \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\" (UID: \"1d3affa8-4f80-428f-ac43-8747dde6ba0f\") " Oct 31 13:06:53.461337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.461309 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-openshift-ca" pod "1d3affa8-4f80-428f-ac43-8747dde6ba0f" (UID: "1d3affa8-4f80-428f-ac43-8747dde6ba0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:53.461465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.461448 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:53.462860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.462833 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-xks-ca" pod "1d3affa8-4f80-428f-ac43-8747dde6ba0f" (UID: "1d3affa8-4f80-428f-ac43-8747dde6ba0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:53.462990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.462957 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3affa8-4f80-428f-ac43-8747dde6ba0f-kube-api-access-rc7mz" pod "1d3affa8-4f80-428f-ac43-8747dde6ba0f" (UID: "1d3affa8-4f80-428f-ac43-8747dde6ba0f"). InnerVolumeSpecName "kube-api-access-rc7mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:53.563263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.563215 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3affa8-4f80-428f-ac43-8747dde6ba0f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:53.563263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:53.563259 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7mz\" (UniqueName: \"kubernetes.io/projected/1d3affa8-4f80-428f-ac43-8747dde6ba0f-kube-api-access-rc7mz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:54.161335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:54.161296 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" event={"ID":"1d3affa8-4f80-428f-ac43-8747dde6ba0f","Type":"ContainerDied","Data":"2328aa652c79e39a759ea6f3b6722fe325b5f787dbec6dbe97fd2c9721a5927f"} Oct 31 13:06:54.161548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:54.161331 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4" Oct 31 13:06:54.161548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:54.161356 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2328aa652c79e39a759ea6f3b6722fe325b5f787dbec6dbe97fd2c9721a5927f" Oct 31 13:06:55.312038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.312006 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd"] Oct 31 13:06:55.312611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.312598 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1d3affa8-4f80-428f-ac43-8747dde6ba0f" containerName="revoke-keys" Oct 31 13:06:55.312697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.312612 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3affa8-4f80-428f-ac43-8747dde6ba0f" containerName="revoke-keys" Oct 31 13:06:55.312697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.312641 5074 cpu_manager.go:414] "containerMap: removing container" podUID="44fdcfa4-e254-4672-b2ed-8b052fe65de2" containerName="revoke-keys" Oct 31 13:06:55.312697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.312646 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fdcfa4-e254-4672-b2ed-8b052fe65de2" containerName="revoke-keys" Oct 31 13:06:55.312791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.312743 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3affa8-4f80-428f-ac43-8747dde6ba0f" containerName="revoke-keys" Oct 31 13:06:55.312791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.312754 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fdcfa4-e254-4672-b2ed-8b052fe65de2" containerName="revoke-keys" Oct 31 13:06:55.316593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.316568 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.330234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.324922 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:06:55.330234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.324982 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:06:55.330234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.324998 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:06:55.330234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.325039 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:06:55.330234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.329035 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd"] Oct 31 13:06:55.495286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.495251 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.495497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.495474 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4jj\" (UniqueName: \"kubernetes.io/projected/403f34f4-982c-4676-a7f3-60d405b28b53-kube-api-access-cp4jj\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.495563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.495525 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.597050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.596956 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4jj\" (UniqueName: \"kubernetes.io/projected/403f34f4-982c-4676-a7f3-60d405b28b53-kube-api-access-cp4jj\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.597050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.597024 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.597306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.597152 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.598017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.597973 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.600012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.599988 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.613275 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.613220 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4jj\" (UniqueName: \"kubernetes.io/projected/403f34f4-982c-4676-a7f3-60d405b28b53-kube-api-access-cp4jj\") pod \"maas-api-revoke-sub-ea1e707d-zgzdd\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.633660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.633621 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:55.937731 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:06:55.937694 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403f34f4_982c_4676_a7f3_60d405b28b53.slice/crio-d7f69d04d9b8ebf0c15f80ed77245ad389081248fd8d0741f6983aee6a7b312c WatchSource:0}: Error finding container d7f69d04d9b8ebf0c15f80ed77245ad389081248fd8d0741f6983aee6a7b312c: Status 404 returned error can't find the container with id d7f69d04d9b8ebf0c15f80ed77245ad389081248fd8d0741f6983aee6a7b312c Oct 31 13:06:55.941884 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:55.941862 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd"] Oct 31 13:06:56.008432 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:06:56.008359 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:06:56.183099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.183055 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" event={"ID":"403f34f4-982c-4676-a7f3-60d405b28b53","Type":"ContainerStarted","Data":"d7f69d04d9b8ebf0c15f80ed77245ad389081248fd8d0741f6983aee6a7b312c"} Oct 31 13:06:56.756374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.756346 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:06:56.808871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.808785 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2"] Oct 31 13:06:56.809693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.809669 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" Oct 31 13:06:56.809693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.809692 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" Oct 31 13:06:56.809925 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.809910 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" Oct 31 13:06:56.820922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.820890 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.824595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.824566 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2"] Oct 31 13:06:56.839491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839466 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-socket\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.839623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839558 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-envoy\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.839623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839604 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-kube-api-access-m7wpb\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-kube-api-access-m7wpb\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.839761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839709 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-token\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.839819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839786 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-socket" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.839875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839820 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-certs\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.839875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839857 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-podinfo\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.839973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.839901 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-data\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840025 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istiod-ca-cert\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840060 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-credential-socket\") pod \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\" (UID: \"9f4a5916-fd55-4eb2-bb66-08f1cdd48371\") " Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840303 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840361 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840394 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhb7\" (UniqueName: \"kubernetes.io/projected/3bd98bbf-7512-4bc7-96d4-5f327818b54c-kube-api-access-lnhb7\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840730 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840744 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istiod-ca-cert" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.840986 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-credential-socket" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.841187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.841064 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-certs" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.841756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.841268 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-data" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.844450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.843386 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-token" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:56.844849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.844811 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-podinfo" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:06:56.844849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.844840 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-envoy" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:56.847712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.847136 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-kube-api-access-m7wpb" pod "9f4a5916-fd55-4eb2-bb66-08f1cdd48371" (UID: "9f4a5916-fd55-4eb2-bb66-08f1cdd48371"). InnerVolumeSpecName "kube-api-access-m7wpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:56.941869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.941833 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.942091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.941890 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.942091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.941928 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhb7\" (UniqueName: \"kubernetes.io/projected/3bd98bbf-7512-4bc7-96d4-5f327818b54c-kube-api-access-lnhb7\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.942223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942166 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942188 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942203 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942218 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942232 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942249 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942262 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7wpb\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-kube-api-access-m7wpb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942278 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4a5916-fd55-4eb2-bb66-08f1cdd48371-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:56.942726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.942705 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.944565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.944538 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:56.955859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:56.955832 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhb7\" (UniqueName: \"kubernetes.io/projected/3bd98bbf-7512-4bc7-96d4-5f327818b54c-kube-api-access-lnhb7\") pod \"maas-api-revoke-sub-cd138956-78lr2\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:57.144256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.144180 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:06:57.225728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.225683 5074 generic.go:356] "Generic (PLEG): container finished" podID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerID="3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc" exitCode=0 Oct 31 13:06:57.228812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.228790 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" Oct 31 13:06:57.236650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.231063 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" event={"ID":"9f4a5916-fd55-4eb2-bb66-08f1cdd48371","Type":"ContainerDied","Data":"3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc"} Oct 31 13:06:57.236650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.231129 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7" event={"ID":"9f4a5916-fd55-4eb2-bb66-08f1cdd48371","Type":"ContainerDied","Data":"3338c261d8a381c619050fed361bc9bb3b598cf8d58e8e62d9146ebdb4544d5e"} Oct 31 13:06:57.236650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.231159 5074 scope.go:122] "RemoveContainer" containerID="3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc" Oct 31 13:06:57.244607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.244580 5074 generic.go:356] "Generic (PLEG): container finished" podID="403f34f4-982c-4676-a7f3-60d405b28b53" containerID="9e48c0c7368bb823a69abb98bb20069f0b6e597dc948943cdb4791382da73210" exitCode=0 Oct 31 13:06:57.244761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.244744 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" event={"ID":"403f34f4-982c-4676-a7f3-60d405b28b53","Type":"ContainerDied","Data":"9e48c0c7368bb823a69abb98bb20069f0b6e597dc948943cdb4791382da73210"} Oct 31 13:06:57.261162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.260820 5074 scope.go:122] "RemoveContainer" containerID="02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc" Oct 31 13:06:57.267579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.267512 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7"] Oct 31 13:06:57.290018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.280041 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4989f9-openshift-default-69b486d7c-5ksh7"] Oct 31 13:06:57.290018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.280094 5074 scope.go:122] "RemoveContainer" containerID="3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc" Oct 31 13:06:57.290018 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:06:57.280597 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc\": container with ID starting with 3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc not found: ID does not exist" containerID="3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc" Oct 31 13:06:57.290018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.280678 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc"} err="failed to get container status \"3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc\": rpc error: code = NotFound desc = could not find container \"3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc\": container with ID starting with 3226aeb13a21ea45b67526e7a044ede6f4dfc0b2bee16fecb5fcebb44acff6cc not found: ID does not exist" Oct 31 13:06:57.290018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.280715 5074 scope.go:122] "RemoveContainer" containerID="02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc" Oct 31 13:06:57.290018 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:06:57.281091 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc\": container with ID starting with 02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc not found: ID does not exist" containerID="02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc" Oct 31 13:06:57.290018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.281119 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc"} err="failed to get container status \"02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc\": rpc error: code = NotFound desc = could not find container \"02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc\": container with ID starting with 02892c261fd8256bc1f615661b6bcecce4365b7953d8e08d0f95c5a1618207dc not found: ID does not exist" Oct 31 13:06:57.526196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.526160 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2"] Oct 31 13:06:57.526528 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:06:57.526504 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd98bbf_7512_4bc7_96d4_5f327818b54c.slice/crio-866841ba0914d4a4281cb0aa661d637f1a7269b352ca912cd1661a1a073ad55d WatchSource:0}: Error finding container 866841ba0914d4a4281cb0aa661d637f1a7269b352ca912cd1661a1a073ad55d: Status 404 returned error can't find the container with id 866841ba0914d4a4281cb0aa661d637f1a7269b352ca912cd1661a1a073ad55d Oct 31 13:06:57.939951 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:57.939920 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" path="/var/lib/kubelet/pods/9f4a5916-fd55-4eb2-bb66-08f1cdd48371/volumes" Oct 31 13:06:58.255622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.255588 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" event={"ID":"3bd98bbf-7512-4bc7-96d4-5f327818b54c","Type":"ContainerStarted","Data":"077965c12547aafb404f30ae5ec83fabb9d14bdb875878d0371a638bfc4ea335"} Oct 31 13:06:58.255822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.255657 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" event={"ID":"3bd98bbf-7512-4bc7-96d4-5f327818b54c","Type":"ContainerStarted","Data":"866841ba0914d4a4281cb0aa661d637f1a7269b352ca912cd1661a1a073ad55d"} Oct 31 13:06:58.522009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.521924 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:58.547882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.547822 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" podStartSLOduration=2.547803963 podStartE2EDuration="2.547803963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:58.310881399 +0000 UTC m=+2112.488599383" watchObservedRunningTime="2026-10-31 13:06:58.547803963 +0000 UTC m=+2112.725521946" Oct 31 13:06:58.570744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.570712 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf"] Oct 31 13:06:58.571708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.571685 5074 cpu_manager.go:414] "containerMap: removing container" podUID="403f34f4-982c-4676-a7f3-60d405b28b53" containerName="revoke-keys" Oct 31 13:06:58.571708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.571707 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403f34f4-982c-4676-a7f3-60d405b28b53" containerName="revoke-keys" Oct 31 13:06:58.571891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.571722 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" Oct 31 13:06:58.571891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.571731 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" Oct 31 13:06:58.571993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.571891 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="403f34f4-982c-4676-a7f3-60d405b28b53" containerName="revoke-keys" Oct 31 13:06:58.571993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.571905 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4a5916-fd55-4eb2-bb66-08f1cdd48371" containerName="istio-proxy" Oct 31 13:06:58.577608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.577490 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 13:06:58.578389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.578367 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.583895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.583873 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf"] Oct 31 13:06:58.626261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.626237 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-openshift-ca\") pod \"403f34f4-982c-4676-a7f3-60d405b28b53\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " Oct 31 13:06:58.626377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.626279 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-xks-ca\") pod \"403f34f4-982c-4676-a7f3-60d405b28b53\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " Oct 31 13:06:58.626498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.626482 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403f34f4-982c-4676-a7f3-60d405b28b53-kube-api-access-cp4jj\" (UniqueName: \"kubernetes.io/projected/403f34f4-982c-4676-a7f3-60d405b28b53-kube-api-access-cp4jj\") pod \"403f34f4-982c-4676-a7f3-60d405b28b53\" (UID: \"403f34f4-982c-4676-a7f3-60d405b28b53\") " Oct 31 13:06:58.626686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.626608 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-openshift-ca" pod "403f34f4-982c-4676-a7f3-60d405b28b53" (UID: "403f34f4-982c-4676-a7f3-60d405b28b53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:58.627678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.627100 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.628682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.628627 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403f34f4-982c-4676-a7f3-60d405b28b53-kube-api-access-cp4jj" pod "403f34f4-982c-4676-a7f3-60d405b28b53" (UID: "403f34f4-982c-4676-a7f3-60d405b28b53"). InnerVolumeSpecName "kube-api-access-cp4jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:58.628985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.628964 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-xks-ca" pod "403f34f4-982c-4676-a7f3-60d405b28b53" (UID: "403f34f4-982c-4676-a7f3-60d405b28b53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:58.728057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728022 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istiod-ca-cert\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728112 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-data\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728142 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-envoy\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728157 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-token\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-token\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728228 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-credential-socket\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728265 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-socket\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728303 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-certs\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728346 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-kube-api-access-rtcp7\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-kube-api-access-rtcp7\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728454 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-podinfo\") pod \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\" (UID: \"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4\") " Oct 31 13:06:58.728608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728478 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-data" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:58.728608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728536 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istiod-ca-cert" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:58.728608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728582 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-credential-socket" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:58.728608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728599 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-socket" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:58.728910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728872 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-certs" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:58.728910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.728886 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.729217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729193 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfk5\" (UniqueName: \"kubernetes.io/projected/f233b40c-5b56-40bb-a75d-8d3caca80f0b-kube-api-access-ncfk5\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.729308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729268 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.729518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729496 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.729558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729528 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f34f4-982c-4676-a7f3-60d405b28b53-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.729558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729544 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.729622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729560 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.729622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729576 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.729622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729590 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.729622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.729604 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4jj\" (UniqueName: \"kubernetes.io/projected/403f34f4-982c-4676-a7f3-60d405b28b53-kube-api-access-cp4jj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.730724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.730695 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-envoy" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:58.730801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.730734 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-podinfo" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:06:58.730933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.730915 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-token" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:58.730970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.730941 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-kube-api-access-rtcp7" pod "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" (UID: "06ec2bc5-3f35-4fbe-82f8-9351d6017ba4"). InnerVolumeSpecName "kube-api-access-rtcp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:58.830869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.830799 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfk5\" (UniqueName: \"kubernetes.io/projected/f233b40c-5b56-40bb-a75d-8d3caca80f0b-kube-api-access-ncfk5\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.830869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.830850 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.831031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.830933 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.831065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.831044 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.831107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.831061 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.831107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.831076 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.831107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.831090 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtcp7\" (UniqueName: \"kubernetes.io/projected/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4-kube-api-access-rtcp7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:58.831614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.831587 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.833680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.833660 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.850217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.850185 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfk5\" (UniqueName: \"kubernetes.io/projected/f233b40c-5b56-40bb-a75d-8d3caca80f0b-kube-api-access-ncfk5\") pod \"maas-api-revoke-sub-f48df00c-bt8zf\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:58.895806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:58.895769 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:06:59.186963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.186928 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf"] Oct 31 13:06:59.187404 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:06:59.187185 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf233b40c_5b56_40bb_a75d_8d3caca80f0b.slice/crio-6cb5ab361d8dc6eef3c59f2148ea03424671b35d5038d0b5268241319cef88ae WatchSource:0}: Error finding container 6cb5ab361d8dc6eef3c59f2148ea03424671b35d5038d0b5268241319cef88ae: Status 404 returned error can't find the container with id 6cb5ab361d8dc6eef3c59f2148ea03424671b35d5038d0b5268241319cef88ae Oct 31 13:06:59.265016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.264978 5074 generic.go:356] "Generic (PLEG): container finished" podID="3bd98bbf-7512-4bc7-96d4-5f327818b54c" containerID="077965c12547aafb404f30ae5ec83fabb9d14bdb875878d0371a638bfc4ea335" exitCode=0 Oct 31 13:06:59.265150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.265105 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" event={"ID":"3bd98bbf-7512-4bc7-96d4-5f327818b54c","Type":"ContainerDied","Data":"077965c12547aafb404f30ae5ec83fabb9d14bdb875878d0371a638bfc4ea335"} Oct 31 13:06:59.266568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.266544 5074 generic.go:356] "Generic (PLEG): container finished" podID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerID="39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2" exitCode=0 Oct 31 13:06:59.266682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.266598 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" Oct 31 13:06:59.266746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.266674 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" event={"ID":"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4","Type":"ContainerDied","Data":"39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2"} Oct 31 13:06:59.266746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.266725 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w" event={"ID":"06ec2bc5-3f35-4fbe-82f8-9351d6017ba4","Type":"ContainerDied","Data":"4d461098cd7ec528a6cc19d8970589cb8f4b6b1fc0616fe420d6e938a6d490fe"} Oct 31 13:06:59.266746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.266740 5074 scope.go:122] "RemoveContainer" containerID="39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2" Oct 31 13:06:59.269959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.269886 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" event={"ID":"f233b40c-5b56-40bb-a75d-8d3caca80f0b","Type":"ContainerStarted","Data":"6cb5ab361d8dc6eef3c59f2148ea03424671b35d5038d0b5268241319cef88ae"} Oct 31 13:06:59.274516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.274489 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" event={"ID":"403f34f4-982c-4676-a7f3-60d405b28b53","Type":"ContainerDied","Data":"d7f69d04d9b8ebf0c15f80ed77245ad389081248fd8d0741f6983aee6a7b312c"} Oct 31 13:06:59.274677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.274538 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f69d04d9b8ebf0c15f80ed77245ad389081248fd8d0741f6983aee6a7b312c" Oct 31 13:06:59.274677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.274587 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd" Oct 31 13:06:59.289838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.281246 5074 scope.go:122] "RemoveContainer" containerID="39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2" Oct 31 13:06:59.289838 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:06:59.281537 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2\": container with ID starting with 39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2 not found: ID does not exist" containerID="39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2" Oct 31 13:06:59.289838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.281570 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2"} err="failed to get container status \"39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2\": rpc error: code = NotFound desc = could not find container \"39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2\": container with ID starting with 39200ea3430e886e5eddf35e6d3bc40d9374a02cfcfe2fca6adf1d21c3979ce2 not found: ID does not exist" Oct 31 13:06:59.307035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.307004 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w"] Oct 31 13:06:59.316611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.316581 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-z5n8w"] Oct 31 13:06:59.939230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:06:59.939143 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" path="/var/lib/kubelet/pods/06ec2bc5-3f35-4fbe-82f8-9351d6017ba4/volumes" Oct 31 13:07:00.286470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.286435 5074 generic.go:356] "Generic (PLEG): container finished" podID="f233b40c-5b56-40bb-a75d-8d3caca80f0b" containerID="a93cd24576e2db37f764b8c67e043a2c8d0ba7470d41698709b25143ef5af1a3" exitCode=0 Oct 31 13:07:00.286980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.286540 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" event={"ID":"f233b40c-5b56-40bb-a75d-8d3caca80f0b","Type":"ContainerDied","Data":"a93cd24576e2db37f764b8c67e043a2c8d0ba7470d41698709b25143ef5af1a3"} Oct 31 13:07:00.456558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.456497 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:07:00.557241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.557200 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-xks-ca\") pod \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " Oct 31 13:07:00.557474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.557454 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-openshift-ca\") pod \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " Oct 31 13:07:00.557562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.557553 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd98bbf-7512-4bc7-96d4-5f327818b54c-kube-api-access-lnhb7\" (UniqueName: \"kubernetes.io/projected/3bd98bbf-7512-4bc7-96d4-5f327818b54c-kube-api-access-lnhb7\") pod \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\" (UID: \"3bd98bbf-7512-4bc7-96d4-5f327818b54c\") " Oct 31 13:07:00.558088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.558056 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-openshift-ca" pod "3bd98bbf-7512-4bc7-96d4-5f327818b54c" (UID: "3bd98bbf-7512-4bc7-96d4-5f327818b54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:00.558269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.558243 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:00.565499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.560484 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd98bbf-7512-4bc7-96d4-5f327818b54c-kube-api-access-lnhb7" pod "3bd98bbf-7512-4bc7-96d4-5f327818b54c" (UID: "3bd98bbf-7512-4bc7-96d4-5f327818b54c"). InnerVolumeSpecName "kube-api-access-lnhb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:00.566279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.566246 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-xks-ca" pod "3bd98bbf-7512-4bc7-96d4-5f327818b54c" (UID: "3bd98bbf-7512-4bc7-96d4-5f327818b54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:00.660144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.660110 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhb7\" (UniqueName: \"kubernetes.io/projected/3bd98bbf-7512-4bc7-96d4-5f327818b54c-kube-api-access-lnhb7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:00.660144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:00.660142 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd98bbf-7512-4bc7-96d4-5f327818b54c-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:01.224711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.224677 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv"] Oct 31 13:07:01.225287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.225251 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3bd98bbf-7512-4bc7-96d4-5f327818b54c" containerName="revoke-keys" Oct 31 13:07:01.225287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.225266 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd98bbf-7512-4bc7-96d4-5f327818b54c" containerName="revoke-keys" Oct 31 13:07:01.225435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.225296 5074 cpu_manager.go:414] "containerMap: removing container" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" Oct 31 13:07:01.225435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.225301 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" Oct 31 13:07:01.225435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.225396 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd98bbf-7512-4bc7-96d4-5f327818b54c" containerName="revoke-keys" Oct 31 13:07:01.225435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.225411 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ec2bc5-3f35-4fbe-82f8-9351d6017ba4" containerName="istio-proxy" Oct 31 13:07:01.229444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.229422 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.239575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.239548 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv"] Oct 31 13:07:01.304085 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.304053 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" Oct 31 13:07:01.305750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.305715 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2" event={"ID":"3bd98bbf-7512-4bc7-96d4-5f327818b54c","Type":"ContainerDied","Data":"866841ba0914d4a4281cb0aa661d637f1a7269b352ca912cd1661a1a073ad55d"} Oct 31 13:07:01.305867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.305803 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866841ba0914d4a4281cb0aa661d637f1a7269b352ca912cd1661a1a073ad55d" Oct 31 13:07:01.372554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.372515 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.372701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.372616 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.372814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.372722 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987bh\" (UniqueName: \"kubernetes.io/projected/76353efb-707d-4d1e-9e8d-19d661f6e00e-kube-api-access-987bh\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.474675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.474626 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.474857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.474733 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.474857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.474822 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987bh\" (UniqueName: \"kubernetes.io/projected/76353efb-707d-4d1e-9e8d-19d661f6e00e-kube-api-access-987bh\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.475591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.475566 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.477877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.477849 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.495467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.495415 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987bh\" (UniqueName: \"kubernetes.io/projected/76353efb-707d-4d1e-9e8d-19d661f6e00e-kube-api-access-987bh\") pod \"maas-api-revoke-sub-2d6be858-c7jjv\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.527757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.527725 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:07:01.546284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.546257 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:01.677663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.677416 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f233b40c-5b56-40bb-a75d-8d3caca80f0b-kube-api-access-ncfk5\" (UniqueName: \"kubernetes.io/projected/f233b40c-5b56-40bb-a75d-8d3caca80f0b-kube-api-access-ncfk5\") pod \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " Oct 31 13:07:01.677663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.677476 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-openshift-ca\") pod \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " Oct 31 13:07:01.678113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.678031 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-xks-ca\") pod \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\" (UID: \"f233b40c-5b56-40bb-a75d-8d3caca80f0b\") " Oct 31 13:07:01.679868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.679822 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-openshift-ca" pod "f233b40c-5b56-40bb-a75d-8d3caca80f0b" (UID: "f233b40c-5b56-40bb-a75d-8d3caca80f0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:01.681600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.681561 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-xks-ca" pod "f233b40c-5b56-40bb-a75d-8d3caca80f0b" (UID: "f233b40c-5b56-40bb-a75d-8d3caca80f0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:01.685055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.685031 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f233b40c-5b56-40bb-a75d-8d3caca80f0b-kube-api-access-ncfk5" pod "f233b40c-5b56-40bb-a75d-8d3caca80f0b" (UID: "f233b40c-5b56-40bb-a75d-8d3caca80f0b"). InnerVolumeSpecName "kube-api-access-ncfk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:01.780022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.779948 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:01.780022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.779975 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncfk5\" (UniqueName: \"kubernetes.io/projected/f233b40c-5b56-40bb-a75d-8d3caca80f0b-kube-api-access-ncfk5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:01.780022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.779989 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f233b40c-5b56-40bb-a75d-8d3caca80f0b-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:01.841997 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:01.841969 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76353efb_707d_4d1e_9e8d_19d661f6e00e.slice/crio-c1092cfdee708ab05b3bc471097953083e9698fd98b7910855dd3a84f0216a47 WatchSource:0}: Error finding container c1092cfdee708ab05b3bc471097953083e9698fd98b7910855dd3a84f0216a47: Status 404 returned error can't find the container with id c1092cfdee708ab05b3bc471097953083e9698fd98b7910855dd3a84f0216a47 Oct 31 13:07:01.842801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:01.842769 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv"] Oct 31 13:07:02.318219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:02.318181 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" event={"ID":"f233b40c-5b56-40bb-a75d-8d3caca80f0b","Type":"ContainerDied","Data":"6cb5ab361d8dc6eef3c59f2148ea03424671b35d5038d0b5268241319cef88ae"} Oct 31 13:07:02.318850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:02.318826 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cb5ab361d8dc6eef3c59f2148ea03424671b35d5038d0b5268241319cef88ae" Oct 31 13:07:02.319198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:02.319160 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf" Oct 31 13:07:02.334953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:02.334912 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" event={"ID":"76353efb-707d-4d1e-9e8d-19d661f6e00e","Type":"ContainerStarted","Data":"3057e0235f467d873b1fd18f95f64004acc1107681ec8581835007eb376edf43"} Oct 31 13:07:02.335118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:02.334975 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" event={"ID":"76353efb-707d-4d1e-9e8d-19d661f6e00e","Type":"ContainerStarted","Data":"c1092cfdee708ab05b3bc471097953083e9698fd98b7910855dd3a84f0216a47"} Oct 31 13:07:02.360287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:02.360197 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" podStartSLOduration=1.360183272 podStartE2EDuration="1.360183272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:02.3558509 +0000 UTC m=+2116.533568883" watchObservedRunningTime="2026-10-31 13:07:02.360183272 +0000 UTC m=+2116.537901274" Oct 31 13:07:02.418980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:02.418948 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:07:03.420654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:03.408206 5074 generic.go:356] "Generic (PLEG): container finished" podID="76353efb-707d-4d1e-9e8d-19d661f6e00e" containerID="3057e0235f467d873b1fd18f95f64004acc1107681ec8581835007eb376edf43" exitCode=0 Oct 31 13:07:03.420654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:03.408462 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" event={"ID":"76353efb-707d-4d1e-9e8d-19d661f6e00e","Type":"ContainerDied","Data":"3057e0235f467d873b1fd18f95f64004acc1107681ec8581835007eb376edf43"} Oct 31 13:07:03.799432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:03.799396 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh"] Oct 31 13:07:03.799902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:03.799868 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" containerID="cri-o://01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5" gracePeriod=30 Oct 31 13:07:04.837830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:04.837720 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:04.968524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:04.968469 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:05.059806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.059753 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:05.111983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.094858 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 13:07:05.136120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.136079 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 13:07:05.351678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.326341 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:05.351678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.328876 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 13:07:05.367739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.335894 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 13:07:05.387667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.370762 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:05.389029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.388975 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:05.408767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.408730 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 13:07:05.732566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.729728 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 31 13:07:05.732566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.730193 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 31 13:07:05.842686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.839321 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:07:05.842686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:05.840043 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:07:06.065706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.064104 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:06.126662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.126410 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:07:06.166108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.166063 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:06.166887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.166864 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 13:07:06.197687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.172717 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76353efb-707d-4d1e-9e8d-19d661f6e00e-kube-api-access-987bh\" (UniqueName: \"kubernetes.io/projected/76353efb-707d-4d1e-9e8d-19d661f6e00e-kube-api-access-987bh\") pod \"76353efb-707d-4d1e-9e8d-19d661f6e00e\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " Oct 31 13:07:06.197687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.172817 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-openshift-ca\") pod \"76353efb-707d-4d1e-9e8d-19d661f6e00e\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " Oct 31 13:07:06.197687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.172919 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-xks-ca\") pod \"76353efb-707d-4d1e-9e8d-19d661f6e00e\" (UID: \"76353efb-707d-4d1e-9e8d-19d661f6e00e\") " Oct 31 13:07:06.260013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.204976 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-openshift-ca" pod "76353efb-707d-4d1e-9e8d-19d661f6e00e" (UID: "76353efb-707d-4d1e-9e8d-19d661f6e00e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:06.260013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.246611 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-xks-ca" pod "76353efb-707d-4d1e-9e8d-19d661f6e00e" (UID: "76353efb-707d-4d1e-9e8d-19d661f6e00e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:06.260013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.256774 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 13:07:06.280609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.280576 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:06.281731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.281703 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76353efb-707d-4d1e-9e8d-19d661f6e00e-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:06.312093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.312052 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76353efb-707d-4d1e-9e8d-19d661f6e00e-kube-api-access-987bh" pod "76353efb-707d-4d1e-9e8d-19d661f6e00e" (UID: "76353efb-707d-4d1e-9e8d-19d661f6e00e"). InnerVolumeSpecName "kube-api-access-987bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:06.384235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.384143 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-987bh\" (UniqueName: \"kubernetes.io/projected/76353efb-707d-4d1e-9e8d-19d661f6e00e-kube-api-access-987bh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:06.835783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.835753 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" Oct 31 13:07:06.835928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.835783 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv" event={"ID":"76353efb-707d-4d1e-9e8d-19d661f6e00e","Type":"ContainerDied","Data":"c1092cfdee708ab05b3bc471097953083e9698fd98b7910855dd3a84f0216a47"} Oct 31 13:07:06.835928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:06.835835 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1092cfdee708ab05b3bc471097953083e9698fd98b7910855dd3a84f0216a47" Oct 31 13:07:07.046853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.046820 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv"] Oct 31 13:07:07.047475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.047453 5074 cpu_manager.go:414] "containerMap: removing container" podUID="76353efb-707d-4d1e-9e8d-19d661f6e00e" containerName="revoke-keys" Oct 31 13:07:07.047475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.047467 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76353efb-707d-4d1e-9e8d-19d661f6e00e" containerName="revoke-keys" Oct 31 13:07:07.047552 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.047489 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f233b40c-5b56-40bb-a75d-8d3caca80f0b" containerName="revoke-keys" Oct 31 13:07:07.047552 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.047494 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f233b40c-5b56-40bb-a75d-8d3caca80f0b" containerName="revoke-keys" Oct 31 13:07:07.047661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.047619 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="76353efb-707d-4d1e-9e8d-19d661f6e00e" containerName="revoke-keys" Oct 31 13:07:07.047661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.047645 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f233b40c-5b56-40bb-a75d-8d3caca80f0b" containerName="revoke-keys" Oct 31 13:07:07.051076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.051056 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.058333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.058310 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:07:07.058456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.058403 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:07:07.060361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.060337 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:07:07.060502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.060392 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:07:07.062699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.062676 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv"] Oct 31 13:07:07.093720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.087918 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-97tmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:07:07.093720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.088035 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:07:07.195588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.195551 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgs6\" (UniqueName: \"kubernetes.io/projected/c94373af-1aa7-436d-a22a-167f128496f6-kube-api-access-pqgs6\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.195780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.195608 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.195780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.195743 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.297492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.297453 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgs6\" (UniqueName: \"kubernetes.io/projected/c94373af-1aa7-436d-a22a-167f128496f6-kube-api-access-pqgs6\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.297733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.297524 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.297733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.297580 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.298282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.298257 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.300379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.300351 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.313876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.313817 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgs6\" (UniqueName: \"kubernetes.io/projected/c94373af-1aa7-436d-a22a-167f128496f6-kube-api-access-pqgs6\") pod \"maas-api-revoke-sub-62c792c7-bpnjv\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.368517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.368415 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:07.674228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.674196 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv"] Oct 31 13:07:07.674327 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:07.674302 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94373af_1aa7_436d_a22a_167f128496f6.slice/crio-221c9af7ad360bcbd88adb26ede188ff089c59518983ff1e20e650f06b4baee8 WatchSource:0}: Error finding container 221c9af7ad360bcbd88adb26ede188ff089c59518983ff1e20e650f06b4baee8: Status 404 returned error can't find the container with id 221c9af7ad360bcbd88adb26ede188ff089c59518983ff1e20e650f06b4baee8 Oct 31 13:07:07.845067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:07.845033 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" event={"ID":"c94373af-1aa7-436d-a22a-167f128496f6","Type":"ContainerStarted","Data":"221c9af7ad360bcbd88adb26ede188ff089c59518983ff1e20e650f06b4baee8"} Oct 31 13:07:08.856408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:08.856381 5074 generic.go:356] "Generic (PLEG): container finished" podID="c94373af-1aa7-436d-a22a-167f128496f6" containerID="bac276f6cc9a320e40ebd75835509c24c13bb47ef38657c4e28c3ab5158fe792" exitCode=0 Oct 31 13:07:08.856808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:08.856463 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" event={"ID":"c94373af-1aa7-436d-a22a-167f128496f6","Type":"ContainerDied","Data":"bac276f6cc9a320e40ebd75835509c24c13bb47ef38657c4e28c3ab5158fe792"} Oct 31 13:07:09.107572 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.107511 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 13:07:09.239260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239216 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-credential-socket\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239293 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-envoy\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239322 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-certs\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239352 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-token\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-token\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239409 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-data\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239478 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-kube-api-access-zp8sc\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-kube-api-access-zp8sc\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239509 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-socket\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239554 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istiod-ca-cert\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239598 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-podinfo\") pod \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\" (UID: \"0bd37002-4c0c-4b16-957b-06bc5ed9c447\") " Oct 31 13:07:09.239867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239508 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-credential-socket" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.239867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239673 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-certs" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.239867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239716 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-data" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.240018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.239905 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-socket" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.240060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.240017 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istiod-ca-cert" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:09.240300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.240275 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.240300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.240299 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.240505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.240309 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.240505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.240323 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.240505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.240336 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.241870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.241845 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-kube-api-access-zp8sc" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "kube-api-access-zp8sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:09.241976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.241876 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-token" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:09.242139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.242115 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-envoy" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:09.242139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.242121 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-podinfo" pod "0bd37002-4c0c-4b16-957b-06bc5ed9c447" (UID: "0bd37002-4c0c-4b16-957b-06bc5ed9c447"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:09.343818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.343775 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.343818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.343816 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.344101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.343829 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.344101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.343842 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp8sc\" (UniqueName: \"kubernetes.io/projected/0bd37002-4c0c-4b16-957b-06bc5ed9c447-kube-api-access-zp8sc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:09.865627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.865591 5074 generic.go:356] "Generic (PLEG): container finished" podID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerID="01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5" exitCode=0 Oct 31 13:07:09.866127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.865665 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" Oct 31 13:07:09.866127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.865673 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" event={"ID":"0bd37002-4c0c-4b16-957b-06bc5ed9c447","Type":"ContainerDied","Data":"01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5"} Oct 31 13:07:09.866127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.865732 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh" event={"ID":"0bd37002-4c0c-4b16-957b-06bc5ed9c447","Type":"ContainerDied","Data":"3a817df39ff8c2c33851e27e8611e7eac0a04c62b39ee178a59406251d58f0b3"} Oct 31 13:07:09.866127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.865758 5074 scope.go:122] "RemoveContainer" containerID="01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5" Oct 31 13:07:09.879558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.879540 5074 scope.go:122] "RemoveContainer" containerID="88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59" Oct 31 13:07:09.889894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.889872 5074 scope.go:122] "RemoveContainer" containerID="01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5" Oct 31 13:07:09.890222 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:07:09.890197 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5\": container with ID starting with 01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5 not found: ID does not exist" containerID="01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5" Oct 31 13:07:09.890296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.890233 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5"} err="failed to get container status \"01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5\": rpc error: code = NotFound desc = could not find container \"01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5\": container with ID starting with 01fe34fae0e710249f5ef3dd2b90c76d272eafe3cd2b5484d4f18cdf24c197a5 not found: ID does not exist" Oct 31 13:07:09.890296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.890253 5074 scope.go:122] "RemoveContainer" containerID="88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59" Oct 31 13:07:09.890572 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:07:09.890555 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59\": container with ID starting with 88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59 not found: ID does not exist" containerID="88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59" Oct 31 13:07:09.890645 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.890579 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59"} err="failed to get container status \"88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59\": rpc error: code = NotFound desc = could not find container \"88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59\": container with ID starting with 88cce96bd57b8a42d607eb12d7a2ea95439bf9fb2741a0c4f4774bc60900fb59 not found: ID does not exist" Oct 31 13:07:09.901399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.901374 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh"] Oct 31 13:07:09.907727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.907697 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-97tmh"] Oct 31 13:07:09.943083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:09.941475 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" path="/var/lib/kubelet/pods/0bd37002-4c0c-4b16-957b-06bc5ed9c447/volumes" Oct 31 13:07:10.103039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.103011 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:10.292073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.292033 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-xks-ca\") pod \"c94373af-1aa7-436d-a22a-167f128496f6\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " Oct 31 13:07:10.292403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.292383 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94373af-1aa7-436d-a22a-167f128496f6-kube-api-access-pqgs6\" (UniqueName: \"kubernetes.io/projected/c94373af-1aa7-436d-a22a-167f128496f6-kube-api-access-pqgs6\") pod \"c94373af-1aa7-436d-a22a-167f128496f6\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " Oct 31 13:07:10.305655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.292590 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-openshift-ca\") pod \"c94373af-1aa7-436d-a22a-167f128496f6\" (UID: \"c94373af-1aa7-436d-a22a-167f128496f6\") " Oct 31 13:07:10.345857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.328115 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-xks-ca" pod "c94373af-1aa7-436d-a22a-167f128496f6" (UID: "c94373af-1aa7-436d-a22a-167f128496f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:10.345857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.328967 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-openshift-ca" pod "c94373af-1aa7-436d-a22a-167f128496f6" (UID: "c94373af-1aa7-436d-a22a-167f128496f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:10.350711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.348855 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94373af-1aa7-436d-a22a-167f128496f6-kube-api-access-pqgs6" pod "c94373af-1aa7-436d-a22a-167f128496f6" (UID: "c94373af-1aa7-436d-a22a-167f128496f6"). InnerVolumeSpecName "kube-api-access-pqgs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:10.408521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.408484 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:10.408691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.408524 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqgs6\" (UniqueName: \"kubernetes.io/projected/c94373af-1aa7-436d-a22a-167f128496f6-kube-api-access-pqgs6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:10.408691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.408541 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94373af-1aa7-436d-a22a-167f128496f6-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:10.876748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.876702 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" Oct 31 13:07:10.877162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.876743 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv" event={"ID":"c94373af-1aa7-436d-a22a-167f128496f6","Type":"ContainerDied","Data":"221c9af7ad360bcbd88adb26ede188ff089c59518983ff1e20e650f06b4baee8"} Oct 31 13:07:10.877162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:10.876837 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221c9af7ad360bcbd88adb26ede188ff089c59518983ff1e20e650f06b4baee8" Oct 31 13:07:14.001422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.000604 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965"] Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001547 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001565 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001586 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001591 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001600 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c94373af-1aa7-436d-a22a-167f128496f6" containerName="revoke-keys" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001605 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94373af-1aa7-436d-a22a-167f128496f6" containerName="revoke-keys" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001783 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001804 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd37002-4c0c-4b16-957b-06bc5ed9c447" containerName="istio-proxy" Oct 31 13:07:14.002030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.001815 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94373af-1aa7-436d-a22a-167f128496f6" containerName="revoke-keys" Oct 31 13:07:14.006112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.006090 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.019870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.019844 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:07:14.020077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.020053 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:07:14.021105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.021073 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965"] Oct 31 13:07:14.042511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.042482 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:07:14.043681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.043023 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:07:14.166965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.166933 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.167136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.167008 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg57f\" (UniqueName: \"kubernetes.io/projected/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-kube-api-access-mg57f\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.167136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.167109 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.268610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.268519 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.268833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.268674 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg57f\" (UniqueName: \"kubernetes.io/projected/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-kube-api-access-mg57f\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.268833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.268744 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.269331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.269294 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.271187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.271165 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.281162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.281140 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg57f\" (UniqueName: \"kubernetes.io/projected/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-kube-api-access-mg57f\") pod \"maas-api-revoke-sub-590dc91d-fl965\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.323392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.323356 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:14.610521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.610485 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965"] Oct 31 13:07:14.610834 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:14.610807 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1aa6219_9a15_4b71_81b4_4dca4230d7a7.slice/crio-4911dbd813b8ee68f56e37304f96beb3708f8d03dfb9319de03e9e7d21ee8555 WatchSource:0}: Error finding container 4911dbd813b8ee68f56e37304f96beb3708f8d03dfb9319de03e9e7d21ee8555: Status 404 returned error can't find the container with id 4911dbd813b8ee68f56e37304f96beb3708f8d03dfb9319de03e9e7d21ee8555 Oct 31 13:07:14.974526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:14.974480 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" event={"ID":"f1aa6219-9a15-4b71-81b4-4dca4230d7a7","Type":"ContainerStarted","Data":"4911dbd813b8ee68f56e37304f96beb3708f8d03dfb9319de03e9e7d21ee8555"} Oct 31 13:07:15.987845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:15.987808 5074 generic.go:356] "Generic (PLEG): container finished" podID="f1aa6219-9a15-4b71-81b4-4dca4230d7a7" containerID="ddab2006f9f44d11cb90f7bc4e76ad65e0b0bae923aff3fd9e548f8c758eb9dd" exitCode=0 Oct 31 13:07:15.988307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:15.987910 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" event={"ID":"f1aa6219-9a15-4b71-81b4-4dca4230d7a7","Type":"ContainerDied","Data":"ddab2006f9f44d11cb90f7bc4e76ad65e0b0bae923aff3fd9e548f8c758eb9dd"} Oct 31 13:07:17.177742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.177710 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:17.239056 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.239024 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-openshift-ca\") pod \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " Oct 31 13:07:17.239248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.239171 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-xks-ca\") pod \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " Oct 31 13:07:17.239320 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.239268 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-kube-api-access-mg57f\" (UniqueName: \"kubernetes.io/projected/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-kube-api-access-mg57f\") pod \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\" (UID: \"f1aa6219-9a15-4b71-81b4-4dca4230d7a7\") " Oct 31 13:07:17.239501 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.239446 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-openshift-ca" pod "f1aa6219-9a15-4b71-81b4-4dca4230d7a7" (UID: "f1aa6219-9a15-4b71-81b4-4dca4230d7a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:17.239946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.239921 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:17.241547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.241513 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-xks-ca" pod "f1aa6219-9a15-4b71-81b4-4dca4230d7a7" (UID: "f1aa6219-9a15-4b71-81b4-4dca4230d7a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:17.241733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.241709 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-kube-api-access-mg57f" pod "f1aa6219-9a15-4b71-81b4-4dca4230d7a7" (UID: "f1aa6219-9a15-4b71-81b4-4dca4230d7a7"). InnerVolumeSpecName "kube-api-access-mg57f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:17.341580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.341546 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:17.341580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.341577 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg57f\" (UniqueName: \"kubernetes.io/projected/f1aa6219-9a15-4b71-81b4-4dca4230d7a7-kube-api-access-mg57f\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:17.881089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:17.881052 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp"] Oct 31 13:07:18.009452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:18.009419 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" event={"ID":"f1aa6219-9a15-4b71-81b4-4dca4230d7a7","Type":"ContainerDied","Data":"4911dbd813b8ee68f56e37304f96beb3708f8d03dfb9319de03e9e7d21ee8555"} Oct 31 13:07:18.009622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:18.009470 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4911dbd813b8ee68f56e37304f96beb3708f8d03dfb9319de03e9e7d21ee8555" Oct 31 13:07:18.009622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:18.009531 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965" Oct 31 13:07:20.127124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.127080 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw"] Oct 31 13:07:20.127968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.127948 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f1aa6219-9a15-4b71-81b4-4dca4230d7a7" containerName="revoke-keys" Oct 31 13:07:20.127968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.127969 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1aa6219-9a15-4b71-81b4-4dca4230d7a7" containerName="revoke-keys" Oct 31 13:07:20.128092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.128082 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1aa6219-9a15-4b71-81b4-4dca4230d7a7" containerName="revoke-keys" Oct 31 13:07:20.131842 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.131824 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.141904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.141877 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:07:20.141904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.141901 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:07:20.147895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.147878 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:07:20.147895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.147878 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:07:20.155618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.155598 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw"] Oct 31 13:07:20.181086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.181031 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7csv8\" (UniqueName: \"kubernetes.io/projected/fde3e0c7-8ea8-4442-ad35-4df68f99877b-kube-api-access-7csv8\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.181247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.181110 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.181331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.181308 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.282804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.282768 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7csv8\" (UniqueName: \"kubernetes.io/projected/fde3e0c7-8ea8-4442-ad35-4df68f99877b-kube-api-access-7csv8\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.283014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.282817 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.283014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.282958 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.283709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.283683 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.285463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.285437 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.297296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.297230 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7csv8\" (UniqueName: \"kubernetes.io/projected/fde3e0c7-8ea8-4442-ad35-4df68f99877b-kube-api-access-7csv8\") pod \"maas-api-revoke-sub-0eaa3a86-bc6kw\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.446173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.446092 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:20.759183 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:20.759150 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde3e0c7_8ea8_4442_ad35_4df68f99877b.slice/crio-43bd76e350645217298f1fdcfd4ef2f9fe024ffe10145fff0ef6a397f049346b WatchSource:0}: Error finding container 43bd76e350645217298f1fdcfd4ef2f9fe024ffe10145fff0ef6a397f049346b: Status 404 returned error can't find the container with id 43bd76e350645217298f1fdcfd4ef2f9fe024ffe10145fff0ef6a397f049346b Oct 31 13:07:20.759957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:20.759935 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw"] Oct 31 13:07:21.036140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:21.036065 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" event={"ID":"fde3e0c7-8ea8-4442-ad35-4df68f99877b","Type":"ContainerStarted","Data":"43bd76e350645217298f1fdcfd4ef2f9fe024ffe10145fff0ef6a397f049346b"} Oct 31 13:07:22.045104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:22.045014 5074 generic.go:356] "Generic (PLEG): container finished" podID="fde3e0c7-8ea8-4442-ad35-4df68f99877b" containerID="761d3d2b85d4217d2376ab1426320d611401f7cce1d72ad6af83176e8352e415" exitCode=0 Oct 31 13:07:22.045494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:22.045101 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" event={"ID":"fde3e0c7-8ea8-4442-ad35-4df68f99877b","Type":"ContainerDied","Data":"761d3d2b85d4217d2376ab1426320d611401f7cce1d72ad6af83176e8352e415"} Oct 31 13:07:22.885655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:22.884707 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" podUID="aaabff29-68e0-4ed9-9b61-66ef2bdbf841" containerName="payload-processing" containerID="cri-o://d6e9900c2bdfd977d6069c407f4c6957dbf75855c1e5fff58b7718e9f9eef511" gracePeriod=25 Oct 31 13:07:23.099664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.098125 5074 generic.go:356] "Generic (PLEG): container finished" podID="aaabff29-68e0-4ed9-9b61-66ef2bdbf841" containerID="d6e9900c2bdfd977d6069c407f4c6957dbf75855c1e5fff58b7718e9f9eef511" exitCode=0 Oct 31 13:07:23.099664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.098773 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" event={"ID":"aaabff29-68e0-4ed9-9b61-66ef2bdbf841","Type":"ContainerDied","Data":"d6e9900c2bdfd977d6069c407f4c6957dbf75855c1e5fff58b7718e9f9eef511"} Oct 31 13:07:23.732981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.732909 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:07:23.815137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.815072 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-plugins-config-volume\") pod \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " Oct 31 13:07:23.815336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.815266 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-kube-api-access-wjk4k\" (UniqueName: \"kubernetes.io/projected/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-kube-api-access-wjk4k\") pod \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\" (UID: \"aaabff29-68e0-4ed9-9b61-66ef2bdbf841\") " Oct 31 13:07:23.819161 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.819110 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-plugins-config-volume" pod "aaabff29-68e0-4ed9-9b61-66ef2bdbf841" (UID: "aaabff29-68e0-4ed9-9b61-66ef2bdbf841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:23.835554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.835524 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-kube-api-access-wjk4k" pod "aaabff29-68e0-4ed9-9b61-66ef2bdbf841" (UID: "aaabff29-68e0-4ed9-9b61-66ef2bdbf841"). InnerVolumeSpecName "kube-api-access-wjk4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:23.837051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.837027 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:23.942352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.918269 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde3e0c7-8ea8-4442-ad35-4df68f99877b-kube-api-access-7csv8\" (UniqueName: \"kubernetes.io/projected/fde3e0c7-8ea8-4442-ad35-4df68f99877b-kube-api-access-7csv8\") pod \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " Oct 31 13:07:23.942352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.918372 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-openshift-ca\") pod \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " Oct 31 13:07:23.942352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.918443 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-xks-ca\") pod \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\" (UID: \"fde3e0c7-8ea8-4442-ad35-4df68f99877b\") " Oct 31 13:07:23.942352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.919000 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:23.942352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.919026 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjk4k\" (UniqueName: \"kubernetes.io/projected/aaabff29-68e0-4ed9-9b61-66ef2bdbf841-kube-api-access-wjk4k\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:23.958940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.949807 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-openshift-ca" pod "fde3e0c7-8ea8-4442-ad35-4df68f99877b" (UID: "fde3e0c7-8ea8-4442-ad35-4df68f99877b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:23.973592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.971880 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-xks-ca" pod "fde3e0c7-8ea8-4442-ad35-4df68f99877b" (UID: "fde3e0c7-8ea8-4442-ad35-4df68f99877b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:23.977662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:23.975806 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde3e0c7-8ea8-4442-ad35-4df68f99877b-kube-api-access-7csv8" pod "fde3e0c7-8ea8-4442-ad35-4df68f99877b" (UID: "fde3e0c7-8ea8-4442-ad35-4df68f99877b"). InnerVolumeSpecName "kube-api-access-7csv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:24.021656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.020430 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7csv8\" (UniqueName: \"kubernetes.io/projected/fde3e0c7-8ea8-4442-ad35-4df68f99877b-kube-api-access-7csv8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:24.021656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.020468 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:24.021656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.020486 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde3e0c7-8ea8-4442-ad35-4df68f99877b-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:24.176571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.171955 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" Oct 31 13:07:24.176571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.172606 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp" event={"ID":"aaabff29-68e0-4ed9-9b61-66ef2bdbf841","Type":"ContainerDied","Data":"d5c8094c9f20d6c7da39e0506bb875364aa7a371f2f53846ecb50091fc1338f6"} Oct 31 13:07:24.176571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.172685 5074 scope.go:122] "RemoveContainer" containerID="d6e9900c2bdfd977d6069c407f4c6957dbf75855c1e5fff58b7718e9f9eef511" Oct 31 13:07:24.197692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.191885 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" event={"ID":"fde3e0c7-8ea8-4442-ad35-4df68f99877b","Type":"ContainerDied","Data":"43bd76e350645217298f1fdcfd4ef2f9fe024ffe10145fff0ef6a397f049346b"} Oct 31 13:07:24.197692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.191942 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43bd76e350645217298f1fdcfd4ef2f9fe024ffe10145fff0ef6a397f049346b" Oct 31 13:07:24.197692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.192010 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw" Oct 31 13:07:24.287333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.287299 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp"] Oct 31 13:07:24.311005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:24.310964 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e4420f-7b459dcbff-hq8gp"] Oct 31 13:07:25.078666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.078294 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 13:07:25.132583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.131428 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.132891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.132864 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.221866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.221814 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.274657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.272729 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.274657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.273082 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.274657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.273332 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 31 13:07:25.393759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.390764 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.414106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.414066 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 13:07:25.480407 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.480361 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.485788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.485743 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:25.781688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:25.779740 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:26.004507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.004448 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaabff29-68e0-4ed9-9b61-66ef2bdbf841" path="/var/lib/kubelet/pods/aaabff29-68e0-4ed9-9b61-66ef2bdbf841/volumes" Oct 31 13:07:26.064865 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:07:26.064694 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:07:26.543742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.543700 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn"] Oct 31 13:07:26.545221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.545190 5074 cpu_manager.go:414] "containerMap: removing container" podUID="aaabff29-68e0-4ed9-9b61-66ef2bdbf841" containerName="payload-processing" Oct 31 13:07:26.545386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.545373 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaabff29-68e0-4ed9-9b61-66ef2bdbf841" containerName="payload-processing" Oct 31 13:07:26.545461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.545452 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fde3e0c7-8ea8-4442-ad35-4df68f99877b" containerName="revoke-keys" Oct 31 13:07:26.545538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.545529 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde3e0c7-8ea8-4442-ad35-4df68f99877b" containerName="revoke-keys" Oct 31 13:07:26.545790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.545777 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaabff29-68e0-4ed9-9b61-66ef2bdbf841" containerName="payload-processing" Oct 31 13:07:26.545886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.545876 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde3e0c7-8ea8-4442-ad35-4df68f99877b" containerName="revoke-keys" Oct 31 13:07:26.601134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.591891 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn"] Oct 31 13:07:26.601509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.595742 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.607172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.607144 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:07:26.607843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.607820 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:07:26.608062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.608048 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:07:26.608223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.608211 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:07:26.657726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.657690 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc"] Oct 31 13:07:26.658493 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.658134 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" containerID="cri-o://2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd" gracePeriod=30 Oct 31 13:07:26.680420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.671573 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswm7\" (UniqueName: \"kubernetes.io/projected/f55a0652-b0f6-4029-bcd7-3b5ba99defca-kube-api-access-wswm7\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.680420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.675809 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.680420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.675888 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.795714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.795274 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswm7\" (UniqueName: \"kubernetes.io/projected/f55a0652-b0f6-4029-bcd7-3b5ba99defca-kube-api-access-wswm7\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.795714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.795410 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.795714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.795474 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.800124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.799582 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.830380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.804555 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.830380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.815999 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswm7\" (UniqueName: \"kubernetes.io/projected/f55a0652-b0f6-4029-bcd7-3b5ba99defca-kube-api-access-wswm7\") pod \"maas-api-revoke-sub-335a33c1-czcjn\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:26.923380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:26.923335 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:27.238408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:27.238373 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn"] Oct 31 13:07:27.238563 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:27.238385 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55a0652_b0f6_4029_bcd7_3b5ba99defca.slice/crio-f64dda53f1239a6936e464d1cd0d2837cc7a992e4ae63c935a8eb6c4e12aefee WatchSource:0}: Error finding container f64dda53f1239a6936e464d1cd0d2837cc7a992e4ae63c935a8eb6c4e12aefee: Status 404 returned error can't find the container with id f64dda53f1239a6936e464d1cd0d2837cc7a992e4ae63c935a8eb6c4e12aefee Oct 31 13:07:27.354202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:27.354158 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" event={"ID":"f55a0652-b0f6-4029-bcd7-3b5ba99defca","Type":"ContainerStarted","Data":"f64dda53f1239a6936e464d1cd0d2837cc7a992e4ae63c935a8eb6c4e12aefee"} Oct 31 13:07:28.372668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:28.372331 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" event={"ID":"f55a0652-b0f6-4029-bcd7-3b5ba99defca","Type":"ContainerStarted","Data":"f73d8f08dac958c0fab7c3b21823d93144cd1c41b08b48ff7a128ae0c3686e7e"} Oct 31 13:07:28.403880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:28.402392 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" podStartSLOduration=2.4023704280000002 podStartE2EDuration="2.402370428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:28.398838478 +0000 UTC m=+2142.576556463" watchObservedRunningTime="2026-10-31 13:07:28.402370428 +0000 UTC m=+2142.580088413" Oct 31 13:07:29.386480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:29.386443 5074 generic.go:356] "Generic (PLEG): container finished" podID="f55a0652-b0f6-4029-bcd7-3b5ba99defca" containerID="f73d8f08dac958c0fab7c3b21823d93144cd1c41b08b48ff7a128ae0c3686e7e" exitCode=0 Oct 31 13:07:29.387034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:29.386486 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" event={"ID":"f55a0652-b0f6-4029-bcd7-3b5ba99defca","Type":"ContainerDied","Data":"f73d8f08dac958c0fab7c3b21823d93144cd1c41b08b48ff7a128ae0c3686e7e"} Oct 31 13:07:30.577216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.577184 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:30.695414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.695332 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-openshift-ca\") pod \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " Oct 31 13:07:30.695414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.695398 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-xks-ca\") pod \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " Oct 31 13:07:30.695652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.695528 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55a0652-b0f6-4029-bcd7-3b5ba99defca-kube-api-access-wswm7\" (UniqueName: \"kubernetes.io/projected/f55a0652-b0f6-4029-bcd7-3b5ba99defca-kube-api-access-wswm7\") pod \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\" (UID: \"f55a0652-b0f6-4029-bcd7-3b5ba99defca\") " Oct 31 13:07:30.695821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.695793 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-openshift-ca" pod "f55a0652-b0f6-4029-bcd7-3b5ba99defca" (UID: "f55a0652-b0f6-4029-bcd7-3b5ba99defca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:30.696145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.696122 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:30.697734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.697703 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-xks-ca" pod "f55a0652-b0f6-4029-bcd7-3b5ba99defca" (UID: "f55a0652-b0f6-4029-bcd7-3b5ba99defca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:30.697837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.697756 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55a0652-b0f6-4029-bcd7-3b5ba99defca-kube-api-access-wswm7" pod "f55a0652-b0f6-4029-bcd7-3b5ba99defca" (UID: "f55a0652-b0f6-4029-bcd7-3b5ba99defca"). InnerVolumeSpecName "kube-api-access-wswm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:30.798209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.798173 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wswm7\" (UniqueName: \"kubernetes.io/projected/f55a0652-b0f6-4029-bcd7-3b5ba99defca-kube-api-access-wswm7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:30.798209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:30.798208 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55a0652-b0f6-4029-bcd7-3b5ba99defca-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:31.420490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:31.420460 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" Oct 31 13:07:31.420490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:31.420482 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn" event={"ID":"f55a0652-b0f6-4029-bcd7-3b5ba99defca","Type":"ContainerDied","Data":"f64dda53f1239a6936e464d1cd0d2837cc7a992e4ae63c935a8eb6c4e12aefee"} Oct 31 13:07:31.420769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:31.420530 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64dda53f1239a6936e464d1cd0d2837cc7a992e4ae63c935a8eb6c4e12aefee" Oct 31 13:07:32.036788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.036756 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.132829 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-socket\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.132979 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-credential-socket\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.133013 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istiod-ca-cert\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.133045 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-certs\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.133082 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-podinfo\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.133115 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-kube-api-access-9psmr\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-kube-api-access-9psmr\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.133233 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-data\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.133318 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-token\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-token\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.133359 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-envoy\") pod \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\" (UID: \"a28977cb-b60d-455e-a6f0-01f849d1ef9d\") " Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.137433 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-certs" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.139725 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-credential-socket" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.140079 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-socket" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.140498 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istiod-ca-cert" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:32.141218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.140866 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-data" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:32.218373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.218030 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-podinfo" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:32.221547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.220550 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-kube-api-access-9psmr" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "kube-api-access-9psmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235463 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235499 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235514 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235529 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235544 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235559 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9psmr\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-kube-api-access-9psmr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235573 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.236721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.235725 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-token" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:32.237254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.236991 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-envoy" pod "a28977cb-b60d-455e-a6f0-01f849d1ef9d" (UID: "a28977cb-b60d-455e-a6f0-01f849d1ef9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:32.343428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.343377 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.343702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.343672 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28977cb-b60d-455e-a6f0-01f849d1ef9d-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:32.455111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.455076 5074 generic.go:356] "Generic (PLEG): container finished" podID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerID="2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd" exitCode=0 Oct 31 13:07:32.455476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.455133 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" Oct 31 13:07:32.456233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.455151 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" event={"ID":"a28977cb-b60d-455e-a6f0-01f849d1ef9d","Type":"ContainerDied","Data":"2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd"} Oct 31 13:07:32.456353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.456273 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc" event={"ID":"a28977cb-b60d-455e-a6f0-01f849d1ef9d","Type":"ContainerDied","Data":"ac90dbe310f9718ce7ed3b3d397bc6e1567d2ec8ef595db551236eeb2f7862e7"} Oct 31 13:07:32.456353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.456311 5074 scope.go:122] "RemoveContainer" containerID="2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd" Oct 31 13:07:32.488087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.474738 5074 scope.go:122] "RemoveContainer" containerID="05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1" Oct 31 13:07:32.542874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.542226 5074 scope.go:122] "RemoveContainer" containerID="2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd" Oct 31 13:07:32.543416 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:07:32.543122 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd\": container with ID starting with 2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd not found: ID does not exist" containerID="2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd" Oct 31 13:07:32.543416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.543166 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd"} err="failed to get container status \"2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd\": rpc error: code = NotFound desc = could not find container \"2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd\": container with ID starting with 2990bf64f22956636e92bc7b4eeb47b12cff98664ed7d732e4c7a782368550fd not found: ID does not exist" Oct 31 13:07:32.543416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.543195 5074 scope.go:122] "RemoveContainer" containerID="05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1" Oct 31 13:07:32.543870 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:07:32.543781 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1\": container with ID starting with 05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1 not found: ID does not exist" containerID="05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1" Oct 31 13:07:32.543870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.543817 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1"} err="failed to get container status \"05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1\": rpc error: code = NotFound desc = could not find container \"05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1\": container with ID starting with 05dafcb5984f2ff5e688c2dd9733bcf939e9cadd626d669dda324195accf30f1 not found: ID does not exist" Oct 31 13:07:32.582999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.582953 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc"] Oct 31 13:07:32.589470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:32.589442 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e4420f-openshift-default-68b7949785-vzhlc"] Oct 31 13:07:33.943078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:33.943031 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" path="/var/lib/kubelet/pods/a28977cb-b60d-455e-a6f0-01f849d1ef9d/volumes" Oct 31 13:07:34.070108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:34.070072 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz"] Oct 31 13:07:34.070383 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:34.070352 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" containerID="cri-o://f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868" gracePeriod=30 Oct 31 13:07:36.686600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.686540 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892"] Oct 31 13:07:36.690134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.690098 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f55a0652-b0f6-4029-bcd7-3b5ba99defca" containerName="revoke-keys" Oct 31 13:07:36.690134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.690132 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55a0652-b0f6-4029-bcd7-3b5ba99defca" containerName="revoke-keys" Oct 31 13:07:36.690417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.690192 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" Oct 31 13:07:36.690417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.690215 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" Oct 31 13:07:36.690623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.690449 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" Oct 31 13:07:36.690623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.690476 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55a0652-b0f6-4029-bcd7-3b5ba99defca" containerName="revoke-keys" Oct 31 13:07:36.705758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.705725 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892"] Oct 31 13:07:36.706095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.706029 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.719966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.719934 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:07:36.723407 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.720684 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:07:36.724518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.721865 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:07:36.724689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.722553 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:07:36.759075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.759036 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.759263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.759087 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.759263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.759231 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnv7\" (UniqueName: \"kubernetes.io/projected/a6f31606-0ffe-4b24-98ed-b01f2b49d773-kube-api-access-4cnv7\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.860837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.860802 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.860837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.860838 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.861112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.860971 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnv7\" (UniqueName: \"kubernetes.io/projected/a6f31606-0ffe-4b24-98ed-b01f2b49d773-kube-api-access-4cnv7\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.861534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.861509 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.863228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.863203 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:36.874673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:36.874623 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnv7\" (UniqueName: \"kubernetes.io/projected/a6f31606-0ffe-4b24-98ed-b01f2b49d773-kube-api-access-4cnv7\") pod \"maas-api-revoke-sub-571bb286-qd892\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:37.036230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:37.036197 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:37.323377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:37.323338 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892"] Oct 31 13:07:37.323516 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:37.323381 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f31606_0ffe_4b24_98ed_b01f2b49d773.slice/crio-c7435228c2d3fe7ef261bbfcbc8cc3fe909406f5e9f5af434b531e828fd3983a WatchSource:0}: Error finding container c7435228c2d3fe7ef261bbfcbc8cc3fe909406f5e9f5af434b531e828fd3983a: Status 404 returned error can't find the container with id c7435228c2d3fe7ef261bbfcbc8cc3fe909406f5e9f5af434b531e828fd3983a Oct 31 13:07:37.546835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:37.546797 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" event={"ID":"a6f31606-0ffe-4b24-98ed-b01f2b49d773","Type":"ContainerStarted","Data":"c7435228c2d3fe7ef261bbfcbc8cc3fe909406f5e9f5af434b531e828fd3983a"} Oct 31 13:07:38.355391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.355353 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm"] Oct 31 13:07:38.355983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.355966 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" Oct 31 13:07:38.355983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.355983 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" Oct 31 13:07:38.356133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.356122 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28977cb-b60d-455e-a6f0-01f849d1ef9d" containerName="istio-proxy" Oct 31 13:07:38.359811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.359787 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.372948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.372920 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm"] Oct 31 13:07:38.423710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.423668 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbde43-0ea8-4353-aec0-1520301c889f-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.423866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.423759 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbde43-0ea8-4353-aec0-1520301c889f-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.423866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.423830 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.423866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.423859 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.424060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.423951 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.424060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.423985 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.424060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.424026 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v6wn\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-kube-api-access-2v6wn\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.424154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.424058 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.424154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.424147 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525309 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525353 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525435 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525469 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525507 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6wn\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-kube-api-access-2v6wn\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525541 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525621 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525737 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbde43-0ea8-4353-aec0-1520301c889f-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.525913 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525818 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbde43-0ea8-4353-aec0-1520301c889f-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.526014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.525992 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-data\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.526084 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.526059 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.526142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.526090 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-certs\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.526195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.526163 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-credential-socket\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.526442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.526422 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbde43-0ea8-4353-aec0-1520301c889f-istiod-ca-cert\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.527989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.527962 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-envoy\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.528266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.528246 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbde43-0ea8-4353-aec0-1520301c889f-istio-podinfo\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.537713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.537683 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-istio-token\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.537820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.537783 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v6wn\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-kube-api-access-2v6wn\") pod \"e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.557256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.557228 5074 generic.go:356] "Generic (PLEG): container finished" podID="a6f31606-0ffe-4b24-98ed-b01f2b49d773" containerID="c283118cc3a53190cfd205d82b3c04755720c5a0ba147c09eda5976df93110f7" exitCode=0 Oct 31 13:07:38.557399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.557311 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" event={"ID":"a6f31606-0ffe-4b24-98ed-b01f2b49d773","Type":"ContainerDied","Data":"c283118cc3a53190cfd205d82b3c04755720c5a0ba147c09eda5976df93110f7"} Oct 31 13:07:38.677994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.677944 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:38.971831 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:38.971755 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cbde43_0ea8_4353_aec0_1520301c889f.slice/crio-f2ae22948fb9ef9e2ba197f31cd4933743539ba34574d550f37680571988a12d WatchSource:0}: Error finding container f2ae22948fb9ef9e2ba197f31cd4933743539ba34574d550f37680571988a12d: Status 404 returned error can't find the container with id f2ae22948fb9ef9e2ba197f31cd4933743539ba34574d550f37680571988a12d Oct 31 13:07:38.975120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.974352 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:38.975120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.974425 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:38.975120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.974459 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:38.980681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:38.980624 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm"] Oct 31 13:07:39.376781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.376647 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 13:07:39.442146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442104 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-data\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442165 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-kube-api-access-jsdp6\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-kube-api-access-jsdp6\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442299 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-token\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-token\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442351 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-certs\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442398 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-envoy\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442437 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdb5206-fc55-415d-bf93-6abad43ae69b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdb5206-fc55-415d-bf93-6abad43ae69b-istiod-ca-cert\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442482 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-podinfo\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442513 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-credential-socket\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.442697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.442569 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-socket\") pod \"9bdb5206-fc55-415d-bf93-6abad43ae69b\" (UID: \"9bdb5206-fc55-415d-bf93-6abad43ae69b\") " Oct 31 13:07:39.443340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.443154 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-certs" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:39.443459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.443364 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-socket" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:39.443459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.443433 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdb5206-fc55-415d-bf93-6abad43ae69b-istiod-ca-cert" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:39.443702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.443682 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-credential-socket" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:39.444149 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.444007 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-data" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:39.446049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.446012 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-podinfo" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:07:39.446360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.446336 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-envoy" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:07:39.446434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.446379 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-token" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:39.446690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.446667 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-kube-api-access-jsdp6" pod "9bdb5206-fc55-415d-bf93-6abad43ae69b" (UID: "9bdb5206-fc55-415d-bf93-6abad43ae69b"). InnerVolumeSpecName "kube-api-access-jsdp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:39.544460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544382 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544414 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsdp6\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-kube-api-access-jsdp6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544424 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544433 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544441 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544451 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdb5206-fc55-415d-bf93-6abad43ae69b-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544460 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdb5206-fc55-415d-bf93-6abad43ae69b-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544468 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.544849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.544477 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdb5206-fc55-415d-bf93-6abad43ae69b-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.566838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.566796 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" event={"ID":"85cbde43-0ea8-4353-aec0-1520301c889f","Type":"ContainerStarted","Data":"8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc"} Oct 31 13:07:39.567036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.566868 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" event={"ID":"85cbde43-0ea8-4353-aec0-1520301c889f","Type":"ContainerStarted","Data":"f2ae22948fb9ef9e2ba197f31cd4933743539ba34574d550f37680571988a12d"} Oct 31 13:07:39.568373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.568348 5074 generic.go:356] "Generic (PLEG): container finished" podID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerID="f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868" exitCode=0 Oct 31 13:07:39.568527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.568383 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" Oct 31 13:07:39.568527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.568425 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" event={"ID":"9bdb5206-fc55-415d-bf93-6abad43ae69b","Type":"ContainerDied","Data":"f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868"} Oct 31 13:07:39.568527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.568479 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz" event={"ID":"9bdb5206-fc55-415d-bf93-6abad43ae69b","Type":"ContainerDied","Data":"9bb51e6f0be213e7c49a19d3d8dbf471fa3de151bf6726e3c8d5e1c3b9d7bfab"} Oct 31 13:07:39.568527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.568507 5074 scope.go:122] "RemoveContainer" containerID="f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868" Oct 31 13:07:39.580231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.580212 5074 scope.go:122] "RemoveContainer" containerID="f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868" Oct 31 13:07:39.580542 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:07:39.580517 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868\": container with ID starting with f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868 not found: ID does not exist" containerID="f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868" Oct 31 13:07:39.580644 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.580549 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868"} err="failed to get container status \"f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868\": rpc error: code = NotFound desc = could not find container \"f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868\": container with ID starting with f3263cc10852aed3d173f299770d77a9d104ff6eef4bf10690d38cfd4ad1e868 not found: ID does not exist" Oct 31 13:07:39.596778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.596724 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podStartSLOduration=1.596707845 podStartE2EDuration="1.596707845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:39.59092578 +0000 UTC m=+2153.768643763" watchObservedRunningTime="2026-10-31 13:07:39.596707845 +0000 UTC m=+2153.774425830" Oct 31 13:07:39.617264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.615327 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz"] Oct 31 13:07:39.620452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.620420 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-5wwzz"] Oct 31 13:07:39.684496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.678396 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:39.754489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.754464 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:39.850168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.850080 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f31606-0ffe-4b24-98ed-b01f2b49d773-kube-api-access-4cnv7\" (UniqueName: \"kubernetes.io/projected/a6f31606-0ffe-4b24-98ed-b01f2b49d773-kube-api-access-4cnv7\") pod \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " Oct 31 13:07:39.850168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.850134 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-openshift-ca\") pod \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " Oct 31 13:07:39.850371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.850334 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-xks-ca\") pod \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\" (UID: \"a6f31606-0ffe-4b24-98ed-b01f2b49d773\") " Oct 31 13:07:39.850546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.850521 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-openshift-ca" pod "a6f31606-0ffe-4b24-98ed-b01f2b49d773" (UID: "a6f31606-0ffe-4b24-98ed-b01f2b49d773"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:07:39.850893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.850871 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.852367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.852343 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f31606-0ffe-4b24-98ed-b01f2b49d773-kube-api-access-4cnv7" pod "a6f31606-0ffe-4b24-98ed-b01f2b49d773" (UID: "a6f31606-0ffe-4b24-98ed-b01f2b49d773"). InnerVolumeSpecName "kube-api-access-4cnv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:07:39.852461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.852446 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-xks-ca" pod "a6f31606-0ffe-4b24-98ed-b01f2b49d773" (UID: "a6f31606-0ffe-4b24-98ed-b01f2b49d773"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:07:39.949683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.949615 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" path="/var/lib/kubelet/pods/9bdb5206-fc55-415d-bf93-6abad43ae69b/volumes" Oct 31 13:07:39.952494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.952465 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f31606-0ffe-4b24-98ed-b01f2b49d773-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:39.952667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:39.952498 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnv7\" (UniqueName: \"kubernetes.io/projected/a6f31606-0ffe-4b24-98ed-b01f2b49d773-kube-api-access-4cnv7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:07:40.606254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:40.606220 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" Oct 31 13:07:40.607068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:40.606242 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892" event={"ID":"a6f31606-0ffe-4b24-98ed-b01f2b49d773","Type":"ContainerDied","Data":"c7435228c2d3fe7ef261bbfcbc8cc3fe909406f5e9f5af434b531e828fd3983a"} Oct 31 13:07:40.607252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:40.607235 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7435228c2d3fe7ef261bbfcbc8cc3fe909406f5e9f5af434b531e828fd3983a" Oct 31 13:07:40.691687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:40.691619 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:07:40.691947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:40.691921 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 31 13:07:41.684657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:41.684047 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:07:41.684657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:41.684111 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 31 13:07:42.688656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:42.688017 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:07:42.688656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:42.688085 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 31 13:07:43.683659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:43.682078 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:07:43.683659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:43.682160 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 31 13:07:44.688102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:44.688059 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:07:44.688733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:44.688700 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 31 13:07:44.814507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:44.814458 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:07:45.678815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:45.678775 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:07:45.679025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:45.678848 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 31 13:07:46.734911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:46.734872 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 31 13:07:46.735547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:46.735510 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 31 13:07:47.226937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:47.225716 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:48.696656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:48.688733 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:07:48.696656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:48.688801 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:49.732088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:49.732036 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:07:49.732667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:49.732106 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:07:49.982957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:49.982858 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:49.983514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:49.983477 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:49.984744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:49.984705 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:07:51.706621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.706582 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9"] Oct 31 13:07:51.707502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.707476 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" Oct 31 13:07:51.707597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.707504 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" Oct 31 13:07:51.707597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.707540 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a6f31606-0ffe-4b24-98ed-b01f2b49d773" containerName="revoke-keys" Oct 31 13:07:51.707597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.707548 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f31606-0ffe-4b24-98ed-b01f2b49d773" containerName="revoke-keys" Oct 31 13:07:51.707811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.707724 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f31606-0ffe-4b24-98ed-b01f2b49d773" containerName="revoke-keys" Oct 31 13:07:51.707811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.707753 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdb5206-fc55-415d-bf93-6abad43ae69b" containerName="istio-proxy" Oct 31 13:07:51.736584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.736550 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9"] Oct 31 13:07:51.736796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.736684 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.743773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.743747 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-5c24e1-openshift-default-dockercfg-mqkrw\"" Oct 31 13:07:51.816694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.811308 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nd75\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-kube-api-access-2nd75\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.817018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.816987 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-envoy\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.826948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.826913 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-socket\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.827187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.827169 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-podinfo\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.827329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.827312 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.827457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.827442 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-credential-socket\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.827580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.827564 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-token\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.830235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.830208 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-certs\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.833982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.833946 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-data\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.945184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.945148 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-data\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.945414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.945398 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nd75\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-kube-api-access-2nd75\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.945519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.945509 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-envoy\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.945670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.945621 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-socket\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.945777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.945766 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-podinfo\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.945864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.945854 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.945932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.945923 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-credential-socket\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.946023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.946013 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-token\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.946096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.946087 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-certs\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.947094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.947064 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-certs\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.947252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.947087 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-credential-socket\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.947449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.947426 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-data\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.947888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.947864 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-socket\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.948019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.947994 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.950777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.950709 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-envoy\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.960331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.960081 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-podinfo\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.978475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.978409 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-token\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:51.994891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:51.994825 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nd75\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-kube-api-access-2nd75\") pod \"e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:52.062718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.062683 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:52.396999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.396964 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9"] Oct 31 13:07:52.402826 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:52.402791 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d13e1c_c11e_4d70_8c1b_01c2679a2f27.slice/crio-eb4f97919bdedf2fd98e4baf70b3dd49247b58db6afc6ce74caf651b881a0b3a WatchSource:0}: Error finding container eb4f97919bdedf2fd98e4baf70b3dd49247b58db6afc6ce74caf651b881a0b3a: Status 404 returned error can't find the container with id eb4f97919bdedf2fd98e4baf70b3dd49247b58db6afc6ce74caf651b881a0b3a Oct 31 13:07:52.406288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.406248 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:52.406419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.406339 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:52.406419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.406382 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:07:52.868788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.868730 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" event={"ID":"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27","Type":"ContainerStarted","Data":"096ac549e26d41832da50adde2f199f3d5e1c62cad6a84f3216624b0db57b9b4"} Oct 31 13:07:52.870801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.870772 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" event={"ID":"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27","Type":"ContainerStarted","Data":"eb4f97919bdedf2fd98e4baf70b3dd49247b58db6afc6ce74caf651b881a0b3a"} Oct 31 13:07:52.905661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:52.901644 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podStartSLOduration=1.901605982 podStartE2EDuration="1.901605982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:52.897319755 +0000 UTC m=+2167.075037739" watchObservedRunningTime="2026-10-31 13:07:52.901605982 +0000 UTC m=+2167.079324012" Oct 31 13:07:53.063047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:53.063004 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:53.069691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:53.069625 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:53.879553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:53.879515 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:53.880505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:53.880483 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:07:56.124975 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:07:56.124930 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:07:58.125648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.125586 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx"] Oct 31 13:07:58.135360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.135330 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.153391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.152102 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx"] Oct 31 13:07:58.153391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.152159 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-6106a3-dockercfg-f992v\"" Oct 31 13:07:58.203765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.203724 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6106a3\"" Oct 31 13:07:58.261171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.261130 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddlz\" (UniqueName: \"kubernetes.io/projected/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-kube-api-access-xddlz\") pod \"payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.273447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.273405 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.385709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.375920 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.385709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.376148 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddlz\" (UniqueName: \"kubernetes.io/projected/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-kube-api-access-xddlz\") pod \"payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.385709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.379670 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.423657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.423355 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddlz\" (UniqueName: \"kubernetes.io/projected/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-kube-api-access-xddlz\") pod \"payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.469715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.469675 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:07:58.710689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.710568 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92"] Oct 31 13:07:58.715941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.715918 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:58.727671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.721934 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-5c24e1\"" Oct 31 13:07:58.727671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.722989 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-5c24e1-dockercfg-dn8x4\"" Oct 31 13:07:58.733474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.733439 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92"] Oct 31 13:07:58.791885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.771844 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx"] Oct 31 13:07:58.791885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.789064 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47db0401-e110-4eb4-ac8a-37b7c864191e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:58.791885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.789155 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s6q6\" (UniqueName: \"kubernetes.io/projected/47db0401-e110-4eb4-ac8a-37b7c864191e-kube-api-access-5s6q6\") pod \"payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:58.891676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.891589 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47db0401-e110-4eb4-ac8a-37b7c864191e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:58.891934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.891743 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s6q6\" (UniqueName: \"kubernetes.io/projected/47db0401-e110-4eb4-ac8a-37b7c864191e-kube-api-access-5s6q6\") pod \"payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:58.895653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.893157 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47db0401-e110-4eb4-ac8a-37b7c864191e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:58.923662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.908344 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s6q6\" (UniqueName: \"kubernetes.io/projected/47db0401-e110-4eb4-ac8a-37b7c864191e-kube-api-access-5s6q6\") pod \"payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:58.996418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:58.996380 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" event={"ID":"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117","Type":"ContainerStarted","Data":"83812b028bd0e8dbcddadcd682f9bd0c4f3bafc72652aa171ed62ccf0df2c489"} Oct 31 13:07:59.051622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:59.051588 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:07:59.352696 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:07:59.352664 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47db0401_e110_4eb4_ac8a_37b7c864191e.slice/crio-cf303a34e6f221ffdcb98e6792aea6a4378fe39039b0565bc510543270e68eb8 WatchSource:0}: Error finding container cf303a34e6f221ffdcb98e6792aea6a4378fe39039b0565bc510543270e68eb8: Status 404 returned error can't find the container with id cf303a34e6f221ffdcb98e6792aea6a4378fe39039b0565bc510543270e68eb8 Oct 31 13:07:59.353456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:07:59.353427 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92"] Oct 31 13:08:00.006728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:00.006685 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" event={"ID":"47db0401-e110-4eb4-ac8a-37b7c864191e","Type":"ContainerStarted","Data":"82097e6bb854595822a2abdfe4d04c53c301c6366b8b117f6a442286010b1658"} Oct 31 13:08:00.006944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:00.006756 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" event={"ID":"47db0401-e110-4eb4-ac8a-37b7c864191e","Type":"ContainerStarted","Data":"cf303a34e6f221ffdcb98e6792aea6a4378fe39039b0565bc510543270e68eb8"} Oct 31 13:08:00.006944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:00.006788 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:08:00.008761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:00.008719 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" event={"ID":"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117","Type":"ContainerStarted","Data":"318b4afc69ce80b72de0ba1d07233590fc2d728ab28841cb05674e0f8f0812cf"} Oct 31 13:08:00.008889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:00.008791 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:08:00.042890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:00.042832 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" podStartSLOduration=2.042819212 podStartE2EDuration="2.042819212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:00.040709351 +0000 UTC m=+2174.218427352" watchObservedRunningTime="2026-10-31 13:08:00.042819212 +0000 UTC m=+2174.220537196" Oct 31 13:08:00.068547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:00.068498 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" podStartSLOduration=2.068480394 podStartE2EDuration="2.068480394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:00.065018268 +0000 UTC m=+2174.242736252" watchObservedRunningTime="2026-10-31 13:08:00.068480394 +0000 UTC m=+2174.246198377" Oct 31 13:08:02.344841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.344806 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr"] Oct 31 13:08:02.349110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.349087 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.374201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.374172 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5452fc-openshift-default-dockercfg-vgclq\"" Oct 31 13:08:02.377224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.377192 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr"] Oct 31 13:08:02.466548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.466506 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.466761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.466607 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.466810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.466769 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.466846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.466834 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.466954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.466937 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.466993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.466980 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.467055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.467030 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.467090 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.467081 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.467134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.467122 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86w7\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-kube-api-access-v86w7\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.568553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568503 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.568822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568568 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.568822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568669 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.568822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568706 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.568822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568804 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.569072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568855 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.569072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568898 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.569072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568943 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.569072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.568976 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86w7\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-kube-api-access-v86w7\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.576668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.570125 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istiod-ca-cert\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.578312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.578283 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-credential-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.579555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.579009 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-data\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.584730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.579223 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-socket\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.584876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.579436 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-certs\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.585607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.585587 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-envoy\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.593650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.590262 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-podinfo\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.602298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.602229 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-token\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.602487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.602329 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86w7\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-kube-api-access-v86w7\") pod \"e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:02.667674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:02.665534 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:03.017365 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:08:03.017321 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff51ea1_b218_4ca6_ac95_d81bdabe324f.slice/crio-627aaf5cb2401074197d4aeca6a9ec6f1ea3c4f7a0d4873d4c9a865468a8dda3 WatchSource:0}: Error finding container 627aaf5cb2401074197d4aeca6a9ec6f1ea3c4f7a0d4873d4c9a865468a8dda3: Status 404 returned error can't find the container with id 627aaf5cb2401074197d4aeca6a9ec6f1ea3c4f7a0d4873d4c9a865468a8dda3 Oct 31 13:08:03.019364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:03.019332 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr"] Oct 31 13:08:03.020265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:03.020232 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:03.020365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:03.020295 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:03.020365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:03.020326 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:03.035940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:03.035913 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" event={"ID":"8ff51ea1-b218-4ca6-ac95-d81bdabe324f","Type":"ContainerStarted","Data":"627aaf5cb2401074197d4aeca6a9ec6f1ea3c4f7a0d4873d4c9a865468a8dda3"} Oct 31 13:08:04.044203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:04.044166 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" event={"ID":"8ff51ea1-b218-4ca6-ac95-d81bdabe324f","Type":"ContainerStarted","Data":"af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51"} Oct 31 13:08:04.078414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:04.078361 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" podStartSLOduration=2.078346492 podStartE2EDuration="2.078346492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:04.075356318 +0000 UTC m=+2178.253074304" watchObservedRunningTime="2026-10-31 13:08:04.078346492 +0000 UTC m=+2178.256064539" Oct 31 13:08:04.666819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:04.666781 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:04.672574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:04.672543 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:05.055124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:05.055091 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:05.056078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:05.056051 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:08.576485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.576450 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr"] Oct 31 13:08:08.582116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.582094 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:08.603041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.603011 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5452fc\"" Oct 31 13:08:08.603166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.603071 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5452fc-dockercfg-c8hvg\"" Oct 31 13:08:08.606758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.606072 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr"] Oct 31 13:08:08.633672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.633610 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4724m\" (UniqueName: \"kubernetes.io/projected/d9ab647a-b861-4530-8d48-f9a05832887b-kube-api-access-4724m\") pod \"payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:08.633797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.633772 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab647a-b861-4530-8d48-f9a05832887b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:08.763664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.760538 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab647a-b861-4530-8d48-f9a05832887b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:08.763664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.760808 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4724m\" (UniqueName: \"kubernetes.io/projected/d9ab647a-b861-4530-8d48-f9a05832887b-kube-api-access-4724m\") pod \"payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:08.763664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.762443 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab647a-b861-4530-8d48-f9a05832887b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:08.859270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.853697 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4724m\" (UniqueName: \"kubernetes.io/projected/d9ab647a-b861-4530-8d48-f9a05832887b-kube-api-access-4724m\") pod \"payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:08.897419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:08.897322 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:09.221185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:09.209707 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr"] Oct 31 13:08:09.221185 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:08:09.214275 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ab647a_b861_4530_8d48_f9a05832887b.slice/crio-65487be109e5726c80fd22927cf075bed247bd145488c5c52be676952ae625b1 WatchSource:0}: Error finding container 65487be109e5726c80fd22927cf075bed247bd145488c5c52be676952ae625b1: Status 404 returned error can't find the container with id 65487be109e5726c80fd22927cf075bed247bd145488c5c52be676952ae625b1 Oct 31 13:08:10.211662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:10.199541 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" event={"ID":"d9ab647a-b861-4530-8d48-f9a05832887b","Type":"ContainerStarted","Data":"60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a"} Oct 31 13:08:10.211662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:10.199614 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" event={"ID":"d9ab647a-b861-4530-8d48-f9a05832887b","Type":"ContainerStarted","Data":"65487be109e5726c80fd22927cf075bed247bd145488c5c52be676952ae625b1"} Oct 31 13:08:10.212529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:10.212449 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:11.030916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:11.030881 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:08:11.032667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:11.031799 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:08:11.058469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:11.058407 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" podStartSLOduration=3.058392546 podStartE2EDuration="3.058392546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:10.266855321 +0000 UTC m=+2184.444573342" watchObservedRunningTime="2026-10-31 13:08:11.058392546 +0000 UTC m=+2185.236110589" Oct 31 13:08:21.850745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:21.850698 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk"] Oct 31 13:08:21.858184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:21.858154 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:21.886655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:21.885718 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-7cdec6\"/\"default-dockercfg-zmtxx\"" Oct 31 13:08:21.886655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:21.885882 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-7cdec6\"/\"openshift-service-ca.crt\"" Oct 31 13:08:21.886655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:21.885985 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-7cdec6\"/\"kube-root-ca.crt\"" Oct 31 13:08:21.886655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:21.886408 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk"] Oct 31 13:08:21.965938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:21.965860 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-7cdec6\"/\"auth-test-model-7cdec6-kserve-self-signed-certs\"" Oct 31 13:08:22.044340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.044291 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7zk\" (UniqueName: \"kubernetes.io/projected/207388c7-33e6-4be1-bf98-22c786edc3bb-kube-api-access-9r7zk\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.044617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.044586 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207388c7-33e6-4be1-bf98-22c786edc3bb-tls-certs\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.044875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.044856 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-dshm\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.045037 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.045008 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-tmp-dir\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.045143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.045123 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-home\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.045275 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.045151 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-model-cache\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.150030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.149925 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-home\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.150392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.150367 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-model-cache\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.151159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.151135 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7zk\" (UniqueName: \"kubernetes.io/projected/207388c7-33e6-4be1-bf98-22c786edc3bb-kube-api-access-9r7zk\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.151776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.150825 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-model-cache\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.151926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.150703 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-home\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.151926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.151767 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207388c7-33e6-4be1-bf98-22c786edc3bb-tls-certs\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.152120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.152097 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-dshm\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.152230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.152210 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-tmp-dir\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.152742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.152718 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-tmp-dir\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.155474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.155450 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207388c7-33e6-4be1-bf98-22c786edc3bb-tls-certs\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.159117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.159090 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-dshm\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.167760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.167736 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7zk\" (UniqueName: \"kubernetes.io/projected/207388c7-33e6-4be1-bf98-22c786edc3bb-kube-api-access-9r7zk\") pod \"auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.250871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.250834 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:08:22.298134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.298098 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:22.712172 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:08:22.712143 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod207388c7_33e6_4be1_bf98_22c786edc3bb.slice/crio-62b023da0278cedf8c8421386294562d81ab2b9e705c5d2eb9efdd4ec50b3abb WatchSource:0}: Error finding container 62b023da0278cedf8c8421386294562d81ab2b9e705c5d2eb9efdd4ec50b3abb: Status 404 returned error can't find the container with id 62b023da0278cedf8c8421386294562d81ab2b9e705c5d2eb9efdd4ec50b3abb Oct 31 13:08:22.712544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:22.712518 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk"] Oct 31 13:08:23.412611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:23.412570 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" event={"ID":"207388c7-33e6-4be1-bf98-22c786edc3bb","Type":"ContainerStarted","Data":"b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db"} Oct 31 13:08:23.412611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:23.412618 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" event={"ID":"207388c7-33e6-4be1-bf98-22c786edc3bb","Type":"ContainerStarted","Data":"62b023da0278cedf8c8421386294562d81ab2b9e705c5d2eb9efdd4ec50b3abb"} Oct 31 13:08:23.435649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:23.435586 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" podStartSLOduration=2.4355707730000002 podStartE2EDuration="2.435570773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:23.432294918 +0000 UTC m=+2197.610012904" watchObservedRunningTime="2026-10-31 13:08:23.435570773 +0000 UTC m=+2197.613288778" Oct 31 13:08:24.316364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:24.316318 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:25.984743 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:08:25.984706 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:08:30.814447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:30.814408 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:08:32.320154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:32.317939 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:32.338438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:32.338397 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:32.513159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:32.513124 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:08:39.921185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:39.920800 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:08:39.921185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:39.920879 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:08:43.709568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.707680 5074 generic.go:356] "Generic (PLEG): container finished" podID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerID="096ac549e26d41832da50adde2f199f3d5e1c62cad6a84f3216624b0db57b9b4" exitCode=0 Oct 31 13:08:43.709568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.707755 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" event={"ID":"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27","Type":"ContainerDied","Data":"096ac549e26d41832da50adde2f199f3d5e1c62cad6a84f3216624b0db57b9b4"} Oct 31 13:08:43.709568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.708272 5074 scope.go:122] "RemoveContainer" containerID="096ac549e26d41832da50adde2f199f3d5e1c62cad6a84f3216624b0db57b9b4" Oct 31 13:08:43.710597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.710484 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.710597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.710551 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.710597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.710585 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.718342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.718305 5074 generic.go:356] "Generic (PLEG): container finished" podID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerID="af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51" exitCode=0 Oct 31 13:08:43.718514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.718375 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" event={"ID":"8ff51ea1-b218-4ca6-ac95-d81bdabe324f","Type":"ContainerDied","Data":"af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51"} Oct 31 13:08:43.718811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.718799 5074 scope.go:122] "RemoveContainer" containerID="af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51" Oct 31 13:08:43.720396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.720367 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.720516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.720433 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:43.720516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:43.720465 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:08:44.748919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:44.748874 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" event={"ID":"8ff51ea1-b218-4ca6-ac95-d81bdabe324f","Type":"ContainerStarted","Data":"4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186"} Oct 31 13:08:44.752086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:44.752055 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" event={"ID":"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27","Type":"ContainerStarted","Data":"64f930168240967cc60f1dbba9b2372c71a46c6921926a7ed40ecf9c15c117f5"} Oct 31 13:08:45.062991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:45.062916 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:08:45.063232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:45.063117 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:08:45.063292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:45.063271 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:08:45.666651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:45.666567 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:45.667728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:45.667697 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:08:45.667809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:45.667752 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:08:46.064480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.064289 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:08:46.064480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.064343 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:08:46.365133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.363935 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6"] Oct 31 13:08:46.452656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.451906 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.460656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.457136 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"kube-root-ca.crt\"" Oct 31 13:08:46.460656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.457349 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"default-dockercfg-kh96p\"" Oct 31 13:08:46.460656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.457416 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"openshift-service-ca.crt\"" Oct 31 13:08:46.460656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.457532 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-45bcad\"/\"auth-test-model-45bcad-kserve-self-signed-certs\"" Oct 31 13:08:46.466837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.466807 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6"] Oct 31 13:08:46.650188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.650091 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-dshm\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.656806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.656758 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-model-cache\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.662773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.662737 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfj9d\" (UniqueName: \"kubernetes.io/projected/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-kube-api-access-wfj9d\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.663067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.663047 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-home\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.663309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.663291 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tmp-dir\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.668117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.668090 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tls-certs\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.678702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.668009 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:08:46.678981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.678944 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:08:46.812653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.810966 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-model-cache\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.812653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.811042 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfj9d\" (UniqueName: \"kubernetes.io/projected/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-kube-api-access-wfj9d\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.812653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.811121 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-home\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.812653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.811232 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tmp-dir\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.812653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.811277 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tls-certs\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.812653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.811404 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-dshm\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.821674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.819033 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-home\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.821674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.819295 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-model-cache\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.821674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.819870 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tmp-dir\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.887893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.879350 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfj9d\" (UniqueName: \"kubernetes.io/projected/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-kube-api-access-wfj9d\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.914699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.913267 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tls-certs\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:46.914699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:46.913581 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-dshm\") pod \"auth-test-model-45bcad-kserve-5ddddf784f-2gtw6\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:47.063052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:47.063016 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:08:47.063255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:47.063063 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:08:47.134081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:47.134017 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:47.667008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:47.666934 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:08:47.667008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:47.666991 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:08:47.739105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:47.739069 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6"] Oct 31 13:08:47.739668 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:08:47.739600 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffcfbe11_91d1_4ec9_8ee6_697bfc85d10b.slice/crio-e9b3d6dc4af0b99bc946c60a3dee216426d7f6a1ba3b068bbcd0c47b26af90aa WatchSource:0}: Error finding container e9b3d6dc4af0b99bc946c60a3dee216426d7f6a1ba3b068bbcd0c47b26af90aa: Status 404 returned error can't find the container with id e9b3d6dc4af0b99bc946c60a3dee216426d7f6a1ba3b068bbcd0c47b26af90aa Oct 31 13:08:47.891456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:47.891366 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" event={"ID":"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b","Type":"ContainerStarted","Data":"e9b3d6dc4af0b99bc946c60a3dee216426d7f6a1ba3b068bbcd0c47b26af90aa"} Oct 31 13:08:48.063517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:48.063456 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:08:48.063733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:48.063522 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:08:48.667152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:48.667111 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:08:48.667547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:48.667167 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:08:48.938446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:48.938345 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" event={"ID":"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b","Type":"ContainerStarted","Data":"3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2"} Oct 31 13:08:48.962621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:48.962572 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" podStartSLOduration=2.962552021 podStartE2EDuration="2.962552021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:48.960224732 +0000 UTC m=+2223.137942717" watchObservedRunningTime="2026-10-31 13:08:48.962552021 +0000 UTC m=+2223.140270046" Oct 31 13:08:49.063282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:49.063242 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:08:49.063515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:49.063295 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:08:49.142762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:49.142723 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:49.666862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:49.666822 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:08:49.667039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:49.666884 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:08:50.060164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:50.060067 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:50.063909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:50.063879 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:08:50.064037 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:50.063949 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:08:50.671649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:50.671607 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:50.978073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:50.978046 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:08:51.071118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:51.071079 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:08:51.071679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:51.071655 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:08:51.072736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:51.072712 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:08:55.983209 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:08:55.983167 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:08:57.136652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:57.134238 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:57.148420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:57.148394 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:58.072877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:58.072843 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:08:59.773373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:08:59.773336 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:09:10.797026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.796977 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k"] Oct 31 13:09:10.801335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.801306 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.818149 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.818115 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-7cdec6-openshift-default-dockercfg-85mvx\"" Oct 31 13:09:10.951845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.951796 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.952049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.951959 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.953680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.952116 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp84k\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-kube-api-access-wp84k\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.953680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.952191 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.953680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.952246 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.953680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.952294 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.953680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.952393 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.953680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.952444 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.953680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.952549 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:10.954841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:10.954821 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k"] Oct 31 13:09:11.055116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.055017 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp84k\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-kube-api-access-wp84k\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059091 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059284 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059335 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059446 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059502 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059662 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059698 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.059867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.059778 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.060268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.060246 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-data\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.061132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.061101 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.062769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.061768 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istiod-ca-cert\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.062769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.062410 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-credential-socket\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.064064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.064041 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-certs\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.080118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.079916 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-envoy\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.092201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.091714 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-podinfo\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.092201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.092155 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-token\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.093060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.093038 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp84k\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-kube-api-access-wp84k\") pod \"e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.287799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.287767 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:11.580057 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:09:11.580025 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391f2b95_3fd5_41cb_867d_74f0bfa8cc2e.slice/crio-758e2488a08939ac3593e5b8f3e9466b2d59f9fc7dc0dc6a684483ca3b01cc46 WatchSource:0}: Error finding container 758e2488a08939ac3593e5b8f3e9466b2d59f9fc7dc0dc6a684483ca3b01cc46: Status 404 returned error can't find the container with id 758e2488a08939ac3593e5b8f3e9466b2d59f9fc7dc0dc6a684483ca3b01cc46 Oct 31 13:09:11.581101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.581078 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k"] Oct 31 13:09:11.582786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.582756 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:11.582866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.582838 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:11.582905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:11.582867 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:12.210697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:12.209844 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" event={"ID":"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e","Type":"ContainerStarted","Data":"df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6"} Oct 31 13:09:12.210697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:12.209916 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" event={"ID":"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e","Type":"ContainerStarted","Data":"758e2488a08939ac3593e5b8f3e9466b2d59f9fc7dc0dc6a684483ca3b01cc46"} Oct 31 13:09:12.241250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:12.241200 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podStartSLOduration=2.2411844690000002 podStartE2EDuration="2.241184469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:12.236368723 +0000 UTC m=+2246.414086713" watchObservedRunningTime="2026-10-31 13:09:12.241184469 +0000 UTC m=+2246.418902454" Oct 31 13:09:12.291653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:12.291371 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:12.303650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:12.298784 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:09:12.303650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:12.298862 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:09:13.288997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:13.288959 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:09:13.290430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:13.290396 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:09:14.307215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:14.307173 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:09:14.307836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:14.307804 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:09:15.294657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:15.294031 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:09:15.294657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:15.294113 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:09:16.293577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:16.293535 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:09:16.294202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:16.294173 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:09:17.289349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:17.289313 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:09:17.289665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:17.289610 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:09:18.296518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:18.296479 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:18.297243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:18.297223 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:18.298439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:18.298416 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:09:19.921589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:19.921552 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2"] Oct 31 13:09:19.928742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:19.927675 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:19.936701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:19.935623 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"kube-root-ca.crt\"" Oct 31 13:09:19.936701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:19.935766 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"openshift-service-ca.crt\"" Oct 31 13:09:19.947258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:19.946748 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"default-dockercfg-hjcmz\"" Oct 31 13:09:19.947258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:19.946944 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"auth-test-model-5452fc-kserve-self-signed-certs\"" Oct 31 13:09:19.949127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:19.948704 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2"] Oct 31 13:09:20.084657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.082719 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2e4a60-039e-4fef-af10-0ece9707bd61-tls-certs\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.084657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.082860 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-model-cache\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.084657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.082896 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzwz\" (UniqueName: \"kubernetes.io/projected/0c2e4a60-039e-4fef-af10-0ece9707bd61-kube-api-access-tnzwz\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.084657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.082935 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-dshm\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.084657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.082964 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-tmp-dir\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.084657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.083019 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-home\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.185438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.185341 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-home\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.185682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.185655 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2e4a60-039e-4fef-af10-0ece9707bd61-tls-certs\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.185784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.185768 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-model-cache\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.185845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.185801 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzwz\" (UniqueName: \"kubernetes.io/projected/0c2e4a60-039e-4fef-af10-0ece9707bd61-kube-api-access-tnzwz\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.185903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.185846 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-dshm\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.185903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.185873 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-tmp-dir\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.186361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.186335 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-tmp-dir\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.186585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.186564 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-home\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.187663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.187598 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-model-cache\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.190873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.190847 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-dshm\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.219686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.216996 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2e4a60-039e-4fef-af10-0ece9707bd61-tls-certs\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.229651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.229582 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzwz\" (UniqueName: \"kubernetes.io/projected/0c2e4a60-039e-4fef-af10-0ece9707bd61-kube-api-access-tnzwz\") pod \"auth-test-model-5452fc-kserve-7db9ff9785-klhs2\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.261369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.261337 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:20.639405 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:09:20.639369 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c2e4a60_039e_4fef_af10_0ece9707bd61.slice/crio-9c458de2c032af9d7b71f9f7fee8bdcf60e0080466fb662ac5374766ec09d9dd WatchSource:0}: Error finding container 9c458de2c032af9d7b71f9f7fee8bdcf60e0080466fb662ac5374766ec09d9dd: Status 404 returned error can't find the container with id 9c458de2c032af9d7b71f9f7fee8bdcf60e0080466fb662ac5374766ec09d9dd Oct 31 13:09:20.640972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:20.640940 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2"] Oct 31 13:09:21.354670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:21.354619 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" event={"ID":"0c2e4a60-039e-4fef-af10-0ece9707bd61","Type":"ContainerStarted","Data":"7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40"} Oct 31 13:09:21.355047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:21.354683 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" event={"ID":"0c2e4a60-039e-4fef-af10-0ece9707bd61","Type":"ContainerStarted","Data":"9c458de2c032af9d7b71f9f7fee8bdcf60e0080466fb662ac5374766ec09d9dd"} Oct 31 13:09:21.383416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:21.383366 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" podStartSLOduration=2.383351554 podStartE2EDuration="2.383351554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:21.381816042 +0000 UTC m=+2255.559534026" watchObservedRunningTime="2026-10-31 13:09:21.383351554 +0000 UTC m=+2255.561069567" Oct 31 13:09:22.266052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:22.266019 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:26.044516 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:09:26.044475 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:09:30.265744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:30.265701 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:30.279465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:30.279438 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:30.451502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:30.451450 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:09:36.393557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:36.393521 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx"] Oct 31 13:09:41.413287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.413241 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" podUID="6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" containerName="payload-pre-processing" containerID="cri-o://318b4afc69ce80b72de0ba1d07233590fc2d728ab28841cb05674e0f8f0812cf" gracePeriod=25 Oct 31 13:09:41.555538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.555500 5074 generic.go:356] "Generic (PLEG): container finished" podID="6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" containerID="318b4afc69ce80b72de0ba1d07233590fc2d728ab28841cb05674e0f8f0812cf" exitCode=0 Oct 31 13:09:41.555698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.555554 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" event={"ID":"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117","Type":"ContainerDied","Data":"318b4afc69ce80b72de0ba1d07233590fc2d728ab28841cb05674e0f8f0812cf"} Oct 31 13:09:41.611864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.611824 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:09:41.718950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.718914 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:09:41.779875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.779834 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-plugins-config-volume\") pod \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " Oct 31 13:09:41.780068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.779946 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-kube-api-access-xddlz\" (UniqueName: \"kubernetes.io/projected/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-kube-api-access-xddlz\") pod \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\" (UID: \"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117\") " Oct 31 13:09:41.780339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.780283 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-plugins-config-volume" pod "6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" (UID: "6e95526d-e8a4-4b53-ad7c-fcb67d6fb117"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:09:41.780689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.780660 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:41.782438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.782410 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-kube-api-access-xddlz" pod "6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" (UID: "6e95526d-e8a4-4b53-ad7c-fcb67d6fb117"). InnerVolumeSpecName "kube-api-access-xddlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:09:41.882220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:41.882181 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xddlz\" (UniqueName: \"kubernetes.io/projected/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117-kube-api-access-xddlz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:09:42.567207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:42.567175 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" Oct 31 13:09:42.567663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:42.567212 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx" event={"ID":"6e95526d-e8a4-4b53-ad7c-fcb67d6fb117","Type":"ContainerDied","Data":"83812b028bd0e8dbcddadcd682f9bd0c4f3bafc72652aa171ed62ccf0df2c489"} Oct 31 13:09:42.567663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:42.567279 5074 scope.go:122] "RemoveContainer" containerID="318b4afc69ce80b72de0ba1d07233590fc2d728ab28841cb05674e0f8f0812cf" Oct 31 13:09:42.616863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:42.616831 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx"] Oct 31 13:09:42.621343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:42.621320 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6106a3-6cccc5488c-2d5fx"] Oct 31 13:09:43.941414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:43.941380 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" path="/var/lib/kubelet/pods/6e95526d-e8a4-4b53-ad7c-fcb67d6fb117/volumes" Oct 31 13:09:51.385967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.385919 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j"] Oct 31 13:09:51.391411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.391376 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" containerName="payload-pre-processing" Oct 31 13:09:51.391670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.391647 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" containerName="payload-pre-processing" Oct 31 13:09:51.392339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.392312 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e95526d-e8a4-4b53-ad7c-fcb67d6fb117" containerName="payload-pre-processing" Oct 31 13:09:51.431033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.430984 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j"] Oct 31 13:09:51.431543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.431521 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.439688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.439661 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-377f60-openshift-default-dockercfg-4cfn6\"" Oct 31 13:09:51.463986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.463950 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-token\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464058 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464163 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-data\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464235 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-podinfo\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464264 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istiod-ca-cert\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464381 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-envoy\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464454 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-certs\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464518 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-credential-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.464654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.464541 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n77wh\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-kube-api-access-n77wh\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566076 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-data\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566142 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-podinfo\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566162 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istiod-ca-cert\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566261 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-envoy\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566309 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-certs\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566432 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-credential-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566468 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n77wh\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-kube-api-access-n77wh\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566536 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-token\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566541 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-data\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566610 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-certs\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566628 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.566950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566926 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istiod-ca-cert\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.567078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.566958 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.567078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.567038 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-credential-socket\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.569151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.569125 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-envoy\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.569370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.569348 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-podinfo\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.581084 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.581055 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n77wh\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-kube-api-access-n77wh\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.581789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.581765 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-token\") pod \"e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:51.762794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:51.762757 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:52.107494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.106815 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j"] Oct 31 13:09:52.112891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.112843 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:52.113012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.112937 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:52.113012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.112982 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:09:52.707998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.707954 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" event={"ID":"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce","Type":"ContainerStarted","Data":"97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570"} Oct 31 13:09:52.708513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.708030 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" event={"ID":"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce","Type":"ContainerStarted","Data":"40935d1e9f28af8d09b4f14dd99b8c3949a242ce849dfe34600992ac01f9b82c"} Oct 31 13:09:52.737825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.737776 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" podStartSLOduration=1.737763428 podStartE2EDuration="1.737763428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:52.735878206 +0000 UTC m=+2286.913596191" watchObservedRunningTime="2026-10-31 13:09:52.737763428 +0000 UTC m=+2286.915481412" Oct 31 13:09:52.763866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:52.763837 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:53.765199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:53.765122 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:09:53.765199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:53.765193 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:09:54.764004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:54.763963 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:09:54.764214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:54.764029 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:09:55.763990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:55.763952 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:09:55.764469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:55.764020 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:09:56.021530 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:09:56.021294 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:09:56.765397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:56.765360 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:09:56.766110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:56.766076 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:09:57.763206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:57.763166 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:09:57.763388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:57.763221 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:09:58.767770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:58.767741 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:58.768235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:58.768161 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:09:58.768922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:09:58.768899 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:10:03.028728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:03.028694 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4"] Oct 31 13:10:03.035590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:03.035557 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988d86e-ccxh4"] Oct 31 13:10:03.941718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:03.941680 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f9d44c-2b1a-4265-9bc8-face74094904" path="/var/lib/kubelet/pods/85f9d44c-2b1a-4265-9bc8-face74094904/volumes" Oct 31 13:10:12.027785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:12.027744 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh"] Oct 31 13:10:12.032019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:12.031996 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88551bca-zcllh"] Oct 31 13:10:13.940393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:13.940359 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da" path="/var/lib/kubelet/pods/ef22136a-2eb8-4ba2-b9c4-dc9e3efab3da/volumes" Oct 31 13:10:16.027152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:16.027115 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4"] Oct 31 13:10:16.032229 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:16.032196 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc881831-fbcv4"] Oct 31 13:10:17.939872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:17.939827 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c4c5db-ef96-405d-9c13-dc90e6bc17e3" path="/var/lib/kubelet/pods/59c4c5db-ef96-405d-9c13-dc90e6bc17e3/volumes" Oct 31 13:10:25.027686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:25.027612 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66"] Oct 31 13:10:25.032300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:25.032274 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea7da3d-ljq66"] Oct 31 13:10:25.942070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:25.942036 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67842144-cce9-4859-b982-d3fea1b70acf" path="/var/lib/kubelet/pods/67842144-cce9-4859-b982-d3fea1b70acf/volumes" Oct 31 13:10:25.987077 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:10:25.987037 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:10:29.026521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:29.026479 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf"] Oct 31 13:10:29.031045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:29.031017 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b278de-4n6qf"] Oct 31 13:10:29.940701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:29.940662 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af15212e-f3f9-486e-b282-1fde1ca23067" path="/var/lib/kubelet/pods/af15212e-f3f9-486e-b282-1fde1ca23067/volumes" Oct 31 13:10:33.905668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:33.905608 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:10:39.046583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:39.046537 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph"] Oct 31 13:10:39.051028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:39.051005 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92f41a62-d9xph"] Oct 31 13:10:39.942204 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:39.942162 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5c5663-bf99-446d-bf67-33498402b507" path="/var/lib/kubelet/pods/ef5c5663-bf99-446d-bf67-33498402b507/volumes" Oct 31 13:10:42.032683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:42.032646 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c"] Oct 31 13:10:42.037029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:42.036996 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b60d0b-k5g7c"] Oct 31 13:10:43.942245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:43.942209 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9eeabb9-f211-49b3-b24f-7c9b3855409a" path="/var/lib/kubelet/pods/f9eeabb9-f211-49b3-b24f-7c9b3855409a/volumes" Oct 31 13:10:52.028361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:52.028321 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9"] Oct 31 13:10:52.032374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:52.032349 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcfa247b-smxw9"] Oct 31 13:10:53.050320 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:53.050283 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2"] Oct 31 13:10:53.065016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:53.064990 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a538cdf-gdrv2"] Oct 31 13:10:53.942384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:53.942331 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610292dc-4e04-4d00-bd33-25a3eac7fd5f" path="/var/lib/kubelet/pods/610292dc-4e04-4d00-bd33-25a3eac7fd5f/volumes" Oct 31 13:10:53.943237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:10:53.943208 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79eeec8b-e321-471c-9e62-722bc9ba45af" path="/var/lib/kubelet/pods/79eeec8b-e321-471c-9e62-722bc9ba45af/volumes" Oct 31 13:10:55.983065 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:10:55.983028 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:11:06.039722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:06.039682 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts"] Oct 31 13:11:06.043598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:06.043569 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462e85ec-gwnts"] Oct 31 13:11:07.940259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:07.940218 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e62ae1b-f87e-465d-aeaa-3b8785a28e03" path="/var/lib/kubelet/pods/1e62ae1b-f87e-465d-aeaa-3b8785a28e03/volumes" Oct 31 13:11:09.027563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:09.027528 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl"] Oct 31 13:11:09.031861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:09.031835 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c873e904-f42bl"] Oct 31 13:11:09.939225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:09.939176 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fada12e-b9a1-4f24-9e7b-8b8a3a41e548" path="/var/lib/kubelet/pods/0fada12e-b9a1-4f24-9e7b-8b8a3a41e548/volumes" Oct 31 13:11:18.025485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:18.025443 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5"] Oct 31 13:11:18.031787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:18.031757 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82fed814-4z5b5"] Oct 31 13:11:20.026980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:20.026942 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24573578-5955-4993-a96b-dc181b2a237e" path="/var/lib/kubelet/pods/24573578-5955-4993-a96b-dc181b2a237e/volumes" Oct 31 13:11:21.509081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:21.509037 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:11:25.984210 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:11:25.984160 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:11:32.025603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:32.025566 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h"] Oct 31 13:11:32.030067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:32.030043 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebdef22-fzj2h"] Oct 31 13:11:33.025544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:33.025508 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf"] Oct 31 13:11:33.029575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:33.029544 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe481ae2-99tkf"] Oct 31 13:11:33.940975 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:33.940941 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e3eea9-248a-4f97-add6-8dc68e27d191" path="/var/lib/kubelet/pods/05e3eea9-248a-4f97-add6-8dc68e27d191/volumes" Oct 31 13:11:33.941436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:33.941421 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2d8c11-b693-4863-8e0d-fb6daac29b0c" path="/var/lib/kubelet/pods/bd2d8c11-b693-4863-8e0d-fb6daac29b0c/volumes" Oct 31 13:11:43.478570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.478531 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p"] Oct 31 13:11:43.482941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.482920 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.508860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.508832 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-45bcad-openshift-default-dockercfg-gfb47\"" Oct 31 13:11:43.512840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.512809 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p"] Oct 31 13:11:43.632440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.632403 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.632667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.632490 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.632667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.632543 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.632790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.632687 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.632790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.632754 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.632909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.632842 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.632970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.632953 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvvt\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-kube-api-access-dwvvt\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.633045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.633023 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.633092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.633069 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.734744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.734663 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.734744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.734707 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.734959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.734758 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvvt\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-kube-api-access-dwvvt\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.734959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.734790 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.734959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.734827 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.734959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.734904 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735004 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735028 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735137 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-certs\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735198 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735411 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-data\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735538 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-credential-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735568 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-socket\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.735717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.735698 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istiod-ca-cert\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.737193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.737172 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-envoy\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.737730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.737712 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-podinfo\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.747455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.747432 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-token\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.747702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.747679 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvvt\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-kube-api-access-dwvvt\") pod \"e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:43.812168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:43.812137 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:44.207675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:44.207300 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p"] Oct 31 13:11:44.208499 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:11:44.208466 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79df00a8_4a3c_4d95_a9e3_f3d8da921927.slice/crio-3df2fb85823156fea50d6ddac6bcb29eb143c09ae538b407ee1000870856e9b1 WatchSource:0}: Error finding container 3df2fb85823156fea50d6ddac6bcb29eb143c09ae538b407ee1000870856e9b1: Status 404 returned error can't find the container with id 3df2fb85823156fea50d6ddac6bcb29eb143c09ae538b407ee1000870856e9b1 Oct 31 13:11:44.212565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:44.212108 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:44.212724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:44.212643 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:44.212724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:44.212691 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:11:44.258945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:44.258918 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" event={"ID":"79df00a8-4a3c-4d95-a9e3-f3d8da921927","Type":"ContainerStarted","Data":"3df2fb85823156fea50d6ddac6bcb29eb143c09ae538b407ee1000870856e9b1"} Oct 31 13:11:45.312419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:45.312375 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" event={"ID":"79df00a8-4a3c-4d95-a9e3-f3d8da921927","Type":"ContainerStarted","Data":"a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320"} Oct 31 13:11:45.813045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:45.813002 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:45.814593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:45.814553 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 13:11:45.814764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:45.814645 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 13:11:46.463893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:46.463864 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:11:46.467898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:46.467873 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:11:46.812541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:46.812509 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 13:11:46.812779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:46.812564 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 13:11:47.813437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:47.813400 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 13:11:47.813971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:47.813466 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 13:11:48.824656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:48.820059 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 13:11:48.824656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:48.820122 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 13:11:49.613524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.613484 5074 scope.go:122] "RemoveContainer" containerID="0c887cee4793cf060b5ec63658e4ab9b29b7008378006e6b837515b74295ed39" Oct 31 13:11:49.702074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.701975 5074 scope.go:122] "RemoveContainer" containerID="9f2461493771d45a97ac8a228b00a37528ead71a4d351c67eb5d095e3dae2884" Oct 31 13:11:49.728031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.728007 5074 scope.go:122] "RemoveContainer" containerID="d3bad3ab37471fca1a95991614312bf345544f9ccc904c7e7046871f0ff53460" Oct 31 13:11:49.738943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.738873 5074 scope.go:122] "RemoveContainer" containerID="4519fa93a3d3caee1e89f8558143c720372b9a530167abafb2ddb2b0b77a3f7d" Oct 31 13:11:49.748511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.748487 5074 scope.go:122] "RemoveContainer" containerID="ca7534fd34dcea97d0551399c690b6d184d66f51d757f12a5b0c9269ace02ea2" Oct 31 13:11:49.762983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.762877 5074 scope.go:122] "RemoveContainer" containerID="0b3ef78e46f53c4a702849bac4bc16a9cbc0c02d8d2463ebdc41502b8413297f" Oct 31 13:11:49.813457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.813275 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 13:11:49.813457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.813327 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 13:11:49.815327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.815301 5074 scope.go:122] "RemoveContainer" containerID="da79e2dafcb0ac4d71d1b36c89ec6e39ab9d950a8cfb791194453b1fb2a62e90" Oct 31 13:11:49.826432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:49.826407 5074 scope.go:122] "RemoveContainer" containerID="8f0d012593ef79354071cc6756da8fd037f07d71ea1c42f8ff774a3175c138bd" Oct 31 13:11:50.812755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:50.812694 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 13:11:50.812950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:50.812796 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 13:11:51.026618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:51.026549 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podStartSLOduration=8.026528026 podStartE2EDuration="8.026528026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:45.339028137 +0000 UTC m=+2399.516746122" watchObservedRunningTime="2026-10-31 13:11:51.026528026 +0000 UTC m=+2405.204246011" Oct 31 13:11:51.028180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:51.028156 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982"] Oct 31 13:11:51.032985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:51.032960 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c3f14dc-lr982"] Oct 31 13:11:51.818591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:51.818560 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:51.819018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:51.818985 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:51.819686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:51.819662 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:11:51.943213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:51.943180 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fdcfa4-e254-4672-b2ed-8b052fe65de2" path="/var/lib/kubelet/pods/44fdcfa4-e254-4672-b2ed-8b052fe65de2/volumes" Oct 31 13:11:53.035623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:53.035585 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4"] Oct 31 13:11:53.051042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:53.051012 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264bc5cd-gqlt4"] Oct 31 13:11:53.940681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:53.940628 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3affa8-4f80-428f-ac43-8747dde6ba0f" path="/var/lib/kubelet/pods/1d3affa8-4f80-428f-ac43-8747dde6ba0f/volumes" Oct 31 13:11:55.309657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:55.304876 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:11:56.148653 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:11:56.142313 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:11:59.025969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.025928 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd"] Oct 31 13:11:59.029703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.029673 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea1e707d-zgzdd"] Oct 31 13:11:59.551118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.551082 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg"] Oct 31 13:11:59.557785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.557755 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.562922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.562892 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:11:59.563853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.563827 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:11:59.564019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.563841 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:11:59.564082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.563844 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:11:59.564611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.564589 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg"] Oct 31 13:11:59.600459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.599939 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.600459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.600088 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.600459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.600174 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrdg\" (UniqueName: \"kubernetes.io/projected/6571282c-39cd-40ae-b384-4a3cf36a7a90-kube-api-access-wzrdg\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.702091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.702052 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.702286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.702122 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrdg\" (UniqueName: \"kubernetes.io/projected/6571282c-39cd-40ae-b384-4a3cf36a7a90-kube-api-access-wzrdg\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.702373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.702351 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.703096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.703077 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.704744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.704720 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.714497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.714469 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrdg\" (UniqueName: \"kubernetes.io/projected/6571282c-39cd-40ae-b384-4a3cf36a7a90-kube-api-access-wzrdg\") pod \"maas-api-revoke-sub-f4156835-c49vg\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.877615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.877533 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:11:59.953818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:11:59.952377 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403f34f4-982c-4676-a7f3-60d405b28b53" path="/var/lib/kubelet/pods/403f34f4-982c-4676-a7f3-60d405b28b53/volumes" Oct 31 13:12:00.199216 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:00.199181 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6571282c_39cd_40ae_b384_4a3cf36a7a90.slice/crio-4b8187746096063f69c334c53495b6499ba332b2cec580f7ab6f24f965566ae6 WatchSource:0}: Error finding container 4b8187746096063f69c334c53495b6499ba332b2cec580f7ab6f24f965566ae6: Status 404 returned error can't find the container with id 4b8187746096063f69c334c53495b6499ba332b2cec580f7ab6f24f965566ae6 Oct 31 13:12:00.200193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:00.200164 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg"] Oct 31 13:12:00.533555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:00.533509 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" event={"ID":"6571282c-39cd-40ae-b384-4a3cf36a7a90","Type":"ContainerStarted","Data":"667642ac50fd1bc3785d587d04ff7904b52b752b0b45cb9ef26be6c153cbb473"} Oct 31 13:12:00.533786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:00.533570 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" event={"ID":"6571282c-39cd-40ae-b384-4a3cf36a7a90","Type":"ContainerStarted","Data":"4b8187746096063f69c334c53495b6499ba332b2cec580f7ab6f24f965566ae6"} Oct 31 13:12:00.559890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:00.559839 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" podStartSLOduration=1.559823414 podStartE2EDuration="1.559823414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:00.556887735 +0000 UTC m=+2414.734605719" watchObservedRunningTime="2026-10-31 13:12:00.559823414 +0000 UTC m=+2414.737541408" Oct 31 13:12:01.030163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:01.030129 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2"] Oct 31 13:12:01.033884 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:01.033853 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd138956-78lr2"] Oct 31 13:12:01.544819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:01.544721 5074 generic.go:356] "Generic (PLEG): container finished" podID="6571282c-39cd-40ae-b384-4a3cf36a7a90" containerID="667642ac50fd1bc3785d587d04ff7904b52b752b0b45cb9ef26be6c153cbb473" exitCode=0 Oct 31 13:12:01.545316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:01.544818 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" event={"ID":"6571282c-39cd-40ae-b384-4a3cf36a7a90","Type":"ContainerDied","Data":"667642ac50fd1bc3785d587d04ff7904b52b752b0b45cb9ef26be6c153cbb473"} Oct 31 13:12:01.940083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:01.939999 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd98bbf-7512-4bc7-96d4-5f327818b54c" path="/var/lib/kubelet/pods/3bd98bbf-7512-4bc7-96d4-5f327818b54c/volumes" Oct 31 13:12:02.029820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.029780 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf"] Oct 31 13:12:02.034265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.034239 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48df00c-bt8zf"] Oct 31 13:12:02.718292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.718266 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:12:02.853019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.852927 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-openshift-ca\") pod \"6571282c-39cd-40ae-b384-4a3cf36a7a90\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " Oct 31 13:12:02.853164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.853088 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6571282c-39cd-40ae-b384-4a3cf36a7a90-kube-api-access-wzrdg\" (UniqueName: \"kubernetes.io/projected/6571282c-39cd-40ae-b384-4a3cf36a7a90-kube-api-access-wzrdg\") pod \"6571282c-39cd-40ae-b384-4a3cf36a7a90\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " Oct 31 13:12:02.853164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.853111 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-xks-ca\") pod \"6571282c-39cd-40ae-b384-4a3cf36a7a90\" (UID: \"6571282c-39cd-40ae-b384-4a3cf36a7a90\") " Oct 31 13:12:02.853296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.853273 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-openshift-ca" pod "6571282c-39cd-40ae-b384-4a3cf36a7a90" (UID: "6571282c-39cd-40ae-b384-4a3cf36a7a90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:02.853648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.853615 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:02.855290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.855264 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-xks-ca" pod "6571282c-39cd-40ae-b384-4a3cf36a7a90" (UID: "6571282c-39cd-40ae-b384-4a3cf36a7a90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:02.855375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.855320 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6571282c-39cd-40ae-b384-4a3cf36a7a90-kube-api-access-wzrdg" pod "6571282c-39cd-40ae-b384-4a3cf36a7a90" (UID: "6571282c-39cd-40ae-b384-4a3cf36a7a90"). InnerVolumeSpecName "kube-api-access-wzrdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:02.954477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.954442 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrdg\" (UniqueName: \"kubernetes.io/projected/6571282c-39cd-40ae-b384-4a3cf36a7a90-kube-api-access-wzrdg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:02.954477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:02.954473 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6571282c-39cd-40ae-b384-4a3cf36a7a90-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:03.564238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:03.564203 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" Oct 31 13:12:03.564397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:03.564241 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg" event={"ID":"6571282c-39cd-40ae-b384-4a3cf36a7a90","Type":"ContainerDied","Data":"4b8187746096063f69c334c53495b6499ba332b2cec580f7ab6f24f965566ae6"} Oct 31 13:12:03.564397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:03.564302 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8187746096063f69c334c53495b6499ba332b2cec580f7ab6f24f965566ae6" Oct 31 13:12:03.947339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:03.947243 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f233b40c-5b56-40bb-a75d-8d3caca80f0b" path="/var/lib/kubelet/pods/f233b40c-5b56-40bb-a75d-8d3caca80f0b/volumes" Oct 31 13:12:07.084120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:07.084074 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv"] Oct 31 13:12:07.089926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:07.089896 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6be858-c7jjv"] Oct 31 13:12:07.939339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:07.939304 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76353efb-707d-4d1e-9e8d-19d661f6e00e" path="/var/lib/kubelet/pods/76353efb-707d-4d1e-9e8d-19d661f6e00e/volumes" Oct 31 13:12:09.820950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.820910 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf"] Oct 31 13:12:09.821907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.821885 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6571282c-39cd-40ae-b384-4a3cf36a7a90" containerName="revoke-keys" Oct 31 13:12:09.821977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.821910 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6571282c-39cd-40ae-b384-4a3cf36a7a90" containerName="revoke-keys" Oct 31 13:12:09.822174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.822161 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6571282c-39cd-40ae-b384-4a3cf36a7a90" containerName="revoke-keys" Oct 31 13:12:09.827803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.827782 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:09.836726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.836625 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:12:09.838417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.837535 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:12:09.838417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.837676 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:12:09.838417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.837703 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf"] Oct 31 13:12:09.838417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.837801 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:12:09.978423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.978375 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:09.978718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.978676 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/0b166cb2-5866-4357-ac29-00be168f8a0e-kube-api-access-qftgd\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:09.978943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:09.978917 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.080710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.080596 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/0b166cb2-5866-4357-ac29-00be168f8a0e-kube-api-access-qftgd\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.081122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.081098 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.081223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.081197 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.081818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.081794 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.083692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.083665 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.093377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.093353 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/0b166cb2-5866-4357-ac29-00be168f8a0e-kube-api-access-qftgd\") pod \"maas-api-revoke-sub-2c54139a-tdsjf\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.145116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.145089 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:10.451987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.451948 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf"] Oct 31 13:12:10.452429 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:10.452402 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b166cb2_5866_4357_ac29_00be168f8a0e.slice/crio-18e2082d1564f7f4203c6e6e6f507d8188996bffacfb05a0184dc3c60ac1a0ac WatchSource:0}: Error finding container 18e2082d1564f7f4203c6e6e6f507d8188996bffacfb05a0184dc3c60ac1a0ac: Status 404 returned error can't find the container with id 18e2082d1564f7f4203c6e6e6f507d8188996bffacfb05a0184dc3c60ac1a0ac Oct 31 13:12:10.630588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:10.630552 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" event={"ID":"0b166cb2-5866-4357-ac29-00be168f8a0e","Type":"ContainerStarted","Data":"18e2082d1564f7f4203c6e6e6f507d8188996bffacfb05a0184dc3c60ac1a0ac"} Oct 31 13:12:11.025340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:11.025305 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv"] Oct 31 13:12:11.028814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:11.028763 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c792c7-bpnjv"] Oct 31 13:12:11.640089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:11.640006 5074 generic.go:356] "Generic (PLEG): container finished" podID="0b166cb2-5866-4357-ac29-00be168f8a0e" containerID="d5371fe9659b219bbd9aa72db434681c77fe63555044be137ca6c3eebc470177" exitCode=0 Oct 31 13:12:11.640244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:11.640082 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" event={"ID":"0b166cb2-5866-4357-ac29-00be168f8a0e","Type":"ContainerDied","Data":"d5371fe9659b219bbd9aa72db434681c77fe63555044be137ca6c3eebc470177"} Oct 31 13:12:11.939434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:11.939357 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94373af-1aa7-436d-a22a-167f128496f6" path="/var/lib/kubelet/pods/c94373af-1aa7-436d-a22a-167f128496f6/volumes" Oct 31 13:12:12.834271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.834245 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:12.865282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.865252 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-xks-ca\") pod \"0b166cb2-5866-4357-ac29-00be168f8a0e\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " Oct 31 13:12:12.865430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.865407 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b166cb2-5866-4357-ac29-00be168f8a0e-kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/0b166cb2-5866-4357-ac29-00be168f8a0e-kube-api-access-qftgd\") pod \"0b166cb2-5866-4357-ac29-00be168f8a0e\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " Oct 31 13:12:12.865666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.865624 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-openshift-ca\") pod \"0b166cb2-5866-4357-ac29-00be168f8a0e\" (UID: \"0b166cb2-5866-4357-ac29-00be168f8a0e\") " Oct 31 13:12:12.866043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.866017 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-openshift-ca" pod "0b166cb2-5866-4357-ac29-00be168f8a0e" (UID: "0b166cb2-5866-4357-ac29-00be168f8a0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:12.866450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.866430 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:12.871260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.871238 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-xks-ca" pod "0b166cb2-5866-4357-ac29-00be168f8a0e" (UID: "0b166cb2-5866-4357-ac29-00be168f8a0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:12.874188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.874139 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b166cb2-5866-4357-ac29-00be168f8a0e-kube-api-access-qftgd" pod "0b166cb2-5866-4357-ac29-00be168f8a0e" (UID: "0b166cb2-5866-4357-ac29-00be168f8a0e"). InnerVolumeSpecName "kube-api-access-qftgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:12.968157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.968126 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b166cb2-5866-4357-ac29-00be168f8a0e-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:12.968157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:12.968155 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/0b166cb2-5866-4357-ac29-00be168f8a0e-kube-api-access-qftgd\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:13.206700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.206658 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs"] Oct 31 13:12:13.207670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.207627 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0b166cb2-5866-4357-ac29-00be168f8a0e" containerName="revoke-keys" Oct 31 13:12:13.207670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.207669 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b166cb2-5866-4357-ac29-00be168f8a0e" containerName="revoke-keys" Oct 31 13:12:13.207890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.207876 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b166cb2-5866-4357-ac29-00be168f8a0e" containerName="revoke-keys" Oct 31 13:12:13.218702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.216045 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs"] Oct 31 13:12:13.218702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.216153 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.227656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.224364 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-842ac9\"" Oct 31 13:12:13.227656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.224483 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-842ac9-dockercfg-vdsbn\"" Oct 31 13:12:13.273057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.273022 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703441e7-a964-49b8-8af3-8c951959bd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.273479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.273456 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm562\" (UniqueName: \"kubernetes.io/projected/703441e7-a964-49b8-8af3-8c951959bd22-kube-api-access-jm562\") pod \"payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.426577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.426089 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703441e7-a964-49b8-8af3-8c951959bd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.426577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.426379 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm562\" (UniqueName: \"kubernetes.io/projected/703441e7-a964-49b8-8af3-8c951959bd22-kube-api-access-jm562\") pod \"payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.427821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.427783 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703441e7-a964-49b8-8af3-8c951959bd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.492570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.492489 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm562\" (UniqueName: \"kubernetes.io/projected/703441e7-a964-49b8-8af3-8c951959bd22-kube-api-access-jm562\") pod \"payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.547658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.537297 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:13.661257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.661226 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" Oct 31 13:12:13.661411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.661255 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf" event={"ID":"0b166cb2-5866-4357-ac29-00be168f8a0e","Type":"ContainerDied","Data":"18e2082d1564f7f4203c6e6e6f507d8188996bffacfb05a0184dc3c60ac1a0ac"} Oct 31 13:12:13.661411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.661309 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e2082d1564f7f4203c6e6e6f507d8188996bffacfb05a0184dc3c60ac1a0ac" Oct 31 13:12:13.848185 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:13.847867 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703441e7_a964_49b8_8af3_8c951959bd22.slice/crio-3cbefd558e1f2b93eb8f6c158187ef2598563732364c1a15d93d729a341e1cd8 WatchSource:0}: Error finding container 3cbefd558e1f2b93eb8f6c158187ef2598563732364c1a15d93d729a341e1cd8: Status 404 returned error can't find the container with id 3cbefd558e1f2b93eb8f6c158187ef2598563732364c1a15d93d729a341e1cd8 Oct 31 13:12:13.849848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:13.849821 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs"] Oct 31 13:12:14.671369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:14.671324 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" event={"ID":"703441e7-a964-49b8-8af3-8c951959bd22","Type":"ContainerStarted","Data":"ba936571746ed2d3be1ca4406bb87f54f5cc676e507401af1d65ed393cf451de"} Oct 31 13:12:14.671680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:14.671428 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" event={"ID":"703441e7-a964-49b8-8af3-8c951959bd22","Type":"ContainerStarted","Data":"3cbefd558e1f2b93eb8f6c158187ef2598563732364c1a15d93d729a341e1cd8"} Oct 31 13:12:14.671680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:14.671568 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:14.692942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:14.692889 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" podStartSLOduration=1.6928770869999998 podStartE2EDuration="1.692877087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:14.689198125 +0000 UTC m=+2428.866916121" watchObservedRunningTime="2026-10-31 13:12:14.692877087 +0000 UTC m=+2428.870595070" Oct 31 13:12:18.063463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:18.063412 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965"] Oct 31 13:12:18.067368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:18.067341 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590dc91d-fl965"] Oct 31 13:12:19.940194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:19.940160 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1aa6219-9a15-4b71-81b4-4dca4230d7a7" path="/var/lib/kubelet/pods/f1aa6219-9a15-4b71-81b4-4dca4230d7a7/volumes" Oct 31 13:12:21.849214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:21.847826 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk"] Oct 31 13:12:24.032253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:24.032215 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw"] Oct 31 13:12:24.036858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:24.036830 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaa3a86-bc6kw"] Oct 31 13:12:25.682857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:25.682822 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:12:25.984585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:25.984551 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde3e0c7-8ea8-4442-ad35-4df68f99877b" path="/var/lib/kubelet/pods/fde3e0c7-8ea8-4442-ad35-4df68f99877b/volumes" Oct 31 13:12:26.021673 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:12:26.021618 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:12:31.056134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:31.056084 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn"] Oct 31 13:12:31.060703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:31.060658 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a33c1-czcjn"] Oct 31 13:12:31.941916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:31.941879 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55a0652-b0f6-4029-bcd7-3b5ba99defca" path="/var/lib/kubelet/pods/f55a0652-b0f6-4029-bcd7-3b5ba99defca/volumes" Oct 31 13:12:32.586356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.586312 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4"] Oct 31 13:12:32.595567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.594092 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.624135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.623500 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9ef744-openshift-default-dockercfg-lbkc7\"" Oct 31 13:12:32.629519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.629492 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4"] Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670509 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istiod-ca-cert\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670670 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-certs\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670772 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-podinfo\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670818 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-socket\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670896 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-token\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670921 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzz97\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-kube-api-access-wzz97\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670953 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-data\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.670995 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-credential-socket\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.671294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.671056 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-envoy\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780005 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-certs\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780143 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-podinfo\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780195 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-socket\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780283 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-token\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780307 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzz97\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-kube-api-access-wzz97\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780341 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-data\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780381 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-credential-socket\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780443 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-envoy\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.780523 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istiod-ca-cert\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.781833 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istiod-ca-cert\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.782133 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-certs\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.785441 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-podinfo\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.785719 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-socket\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.786654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.786432 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-data\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.787552 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.786681 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-credential-socket\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.824106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.820087 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-envoy\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.826303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.826267 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-token\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.852711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.834562 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzz97\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-kube-api-access-wzz97\") pod \"e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:32.945953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:32.945914 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:33.349114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:33.349081 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4"] Oct 31 13:12:33.350853 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:33.350826 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127032f3_bc89_42ff_ae41_c1ff7e9b0edb.slice/crio-f92037417e4f82604d85f5376ebe458ea12990ffbe6616ee690887763511cfb7 WatchSource:0}: Error finding container f92037417e4f82604d85f5376ebe458ea12990ffbe6616ee690887763511cfb7: Status 404 returned error can't find the container with id f92037417e4f82604d85f5376ebe458ea12990ffbe6616ee690887763511cfb7 Oct 31 13:12:33.354262 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:33.354232 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:33.354348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:33.354296 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:33.354348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:33.354338 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:12:33.980726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:33.980691 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" event={"ID":"127032f3-bc89-42ff-ae41-c1ff7e9b0edb","Type":"ContainerStarted","Data":"1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157"} Oct 31 13:12:33.981126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:33.980748 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" event={"ID":"127032f3-bc89-42ff-ae41-c1ff7e9b0edb","Type":"ContainerStarted","Data":"f92037417e4f82604d85f5376ebe458ea12990ffbe6616ee690887763511cfb7"} Oct 31 13:12:34.053458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:34.053377 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" podStartSLOduration=2.053356161 podStartE2EDuration="2.053356161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:34.045750774 +0000 UTC m=+2448.223468760" watchObservedRunningTime="2026-10-31 13:12:34.053356161 +0000 UTC m=+2448.231074145" Oct 31 13:12:34.267939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:34.267846 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:12:34.946536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:34.946506 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:34.952571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:34.952540 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:34.991653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:34.991606 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:34.992397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:34.992377 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:12:36.914745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:36.913816 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" podUID="207388c7-33e6-4be1-bf98-22c786edc3bb" containerName="main" containerID="cri-o://b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db" gracePeriod=45 Oct 31 13:12:37.296181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.296156 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:12:37.323316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323291 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-tmp-dir\") pod \"207388c7-33e6-4be1-bf98-22c786edc3bb\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " Oct 31 13:12:37.323478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323415 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207388c7-33e6-4be1-bf98-22c786edc3bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/207388c7-33e6-4be1-bf98-22c786edc3bb-tls-certs\") pod \"207388c7-33e6-4be1-bf98-22c786edc3bb\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " Oct 31 13:12:37.323478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323461 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-model-cache\") pod \"207388c7-33e6-4be1-bf98-22c786edc3bb\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " Oct 31 13:12:37.323610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323503 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-home\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-home\") pod \"207388c7-33e6-4be1-bf98-22c786edc3bb\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " Oct 31 13:12:37.323610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323530 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-dshm\") pod \"207388c7-33e6-4be1-bf98-22c786edc3bb\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " Oct 31 13:12:37.323610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323560 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207388c7-33e6-4be1-bf98-22c786edc3bb-kube-api-access-9r7zk\" (UniqueName: \"kubernetes.io/projected/207388c7-33e6-4be1-bf98-22c786edc3bb-kube-api-access-9r7zk\") pod \"207388c7-33e6-4be1-bf98-22c786edc3bb\" (UID: \"207388c7-33e6-4be1-bf98-22c786edc3bb\") " Oct 31 13:12:37.323610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323572 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-tmp-dir" pod "207388c7-33e6-4be1-bf98-22c786edc3bb" (UID: "207388c7-33e6-4be1-bf98-22c786edc3bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.323864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323840 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-model-cache" pod "207388c7-33e6-4be1-bf98-22c786edc3bb" (UID: "207388c7-33e6-4be1-bf98-22c786edc3bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.323864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.323848 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-home" pod "207388c7-33e6-4be1-bf98-22c786edc3bb" (UID: "207388c7-33e6-4be1-bf98-22c786edc3bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.324279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.324255 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.324392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.324283 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.324392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.324297 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.326126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.326099 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207388c7-33e6-4be1-bf98-22c786edc3bb-kube-api-access-9r7zk" pod "207388c7-33e6-4be1-bf98-22c786edc3bb" (UID: "207388c7-33e6-4be1-bf98-22c786edc3bb"). InnerVolumeSpecName "kube-api-access-9r7zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:37.326229 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.326125 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-dshm" pod "207388c7-33e6-4be1-bf98-22c786edc3bb" (UID: "207388c7-33e6-4be1-bf98-22c786edc3bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:12:37.326354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.326323 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207388c7-33e6-4be1-bf98-22c786edc3bb-tls-certs" pod "207388c7-33e6-4be1-bf98-22c786edc3bb" (UID: "207388c7-33e6-4be1-bf98-22c786edc3bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:37.432047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.432008 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207388c7-33e6-4be1-bf98-22c786edc3bb-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.432047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.432050 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207388c7-33e6-4be1-bf98-22c786edc3bb-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:37.432308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:37.432066 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r7zk\" (UniqueName: \"kubernetes.io/projected/207388c7-33e6-4be1-bf98-22c786edc3bb-kube-api-access-9r7zk\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:38.056485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.056442 5074 generic.go:356] "Generic (PLEG): container finished" podID="207388c7-33e6-4be1-bf98-22c786edc3bb" containerID="b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db" exitCode=0 Oct 31 13:12:38.057076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.056546 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" Oct 31 13:12:38.057076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.056541 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" event={"ID":"207388c7-33e6-4be1-bf98-22c786edc3bb","Type":"ContainerDied","Data":"b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db"} Oct 31 13:12:38.057076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.056616 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk" event={"ID":"207388c7-33e6-4be1-bf98-22c786edc3bb","Type":"ContainerDied","Data":"62b023da0278cedf8c8421386294562d81ab2b9e705c5d2eb9efdd4ec50b3abb"} Oct 31 13:12:38.057076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.056658 5074 scope.go:122] "RemoveContainer" containerID="b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db" Oct 31 13:12:38.069312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.069284 5074 scope.go:122] "RemoveContainer" containerID="b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db" Oct 31 13:12:38.069694 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:12:38.069670 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db\": container with ID starting with b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db not found: ID does not exist" containerID="b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db" Oct 31 13:12:38.069783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.069703 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db"} err="failed to get container status \"b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db\": rpc error: code = NotFound desc = could not find container \"b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db\": container with ID starting with b5fbecffddf744c1634221c540f1b2eb41c8a1eeb87b28dbdd2d57f3609ce8db not found: ID does not exist" Oct 31 13:12:38.111701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.111659 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk"] Oct 31 13:12:38.126703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.126566 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-7cdec6/auth-test-model-7cdec6-kserve-6f4b949f57-d7lxk"] Oct 31 13:12:38.542095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.542060 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf"] Oct 31 13:12:38.543044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.543024 5074 cpu_manager.go:414] "containerMap: removing container" podUID="207388c7-33e6-4be1-bf98-22c786edc3bb" containerName="main" Oct 31 13:12:38.543104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.543047 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207388c7-33e6-4be1-bf98-22c786edc3bb" containerName="main" Oct 31 13:12:38.543266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.543254 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="207388c7-33e6-4be1-bf98-22c786edc3bb" containerName="main" Oct 31 13:12:38.548761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.548729 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.574421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.574384 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:12:38.574708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.574670 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:12:38.620236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.620045 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:12:38.620236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.620196 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:12:38.691667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.667708 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf"] Oct 31 13:12:38.691667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.669132 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.691667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.669246 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.691667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.669341 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6kr\" (UniqueName: \"kubernetes.io/projected/e268dce8-438c-4f50-b0ad-7253f98a5eeb-kube-api-access-dw6kr\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.794491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.794394 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.795548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.795520 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.795815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.795796 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6kr\" (UniqueName: \"kubernetes.io/projected/e268dce8-438c-4f50-b0ad-7253f98a5eeb-kube-api-access-dw6kr\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.796540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.796498 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.852889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.838137 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.862949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.852958 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9"] Oct 31 13:12:38.862949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.861732 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6kr\" (UniqueName: \"kubernetes.io/projected/e268dce8-438c-4f50-b0ad-7253f98a5eeb-kube-api-access-dw6kr\") pod \"maas-api-revoke-sub-61a32386-mc9pf\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.869657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.865055 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:38.881401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.876852 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9"] Oct 31 13:12:38.881401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.876961 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:38.905723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.904766 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:38.905723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.904930 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jhj\" (UniqueName: \"kubernetes.io/projected/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-kube-api-access-r6jhj\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:38.905723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:38.905148 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.006818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.006783 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jhj\" (UniqueName: \"kubernetes.io/projected/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-kube-api-access-r6jhj\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.007122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.007093 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.007206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.007182 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.009328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.009276 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.010431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.010360 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.023915 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.023855 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jhj\" (UniqueName: \"kubernetes.io/projected/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-kube-api-access-r6jhj\") pod \"maas-api-revoke-sub-7f8657a3-g22r9\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.185674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.185628 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf"] Oct 31 13:12:39.186324 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:39.186289 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode268dce8_438c_4f50_b0ad_7253f98a5eeb.slice/crio-f98f8e64d051c21a4e68318ba5b1eb5aa940f53d064c2aff9fdc36ca8e3c87cd WatchSource:0}: Error finding container f98f8e64d051c21a4e68318ba5b1eb5aa940f53d064c2aff9fdc36ca8e3c87cd: Status 404 returned error can't find the container with id f98f8e64d051c21a4e68318ba5b1eb5aa940f53d064c2aff9fdc36ca8e3c87cd Oct 31 13:12:39.238384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.238360 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:39.641562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.641532 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9"] Oct 31 13:12:39.642128 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:39.642101 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f4ddd7_ccbb_4858_87c3_f7b69dbe9070.slice/crio-e5a31ec797bb8d5e26fd39c20a863a82b41bb438afecf4942cf3de8042800e85 WatchSource:0}: Error finding container e5a31ec797bb8d5e26fd39c20a863a82b41bb438afecf4942cf3de8042800e85: Status 404 returned error can't find the container with id e5a31ec797bb8d5e26fd39c20a863a82b41bb438afecf4942cf3de8042800e85 Oct 31 13:12:39.941560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:39.941523 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207388c7-33e6-4be1-bf98-22c786edc3bb" path="/var/lib/kubelet/pods/207388c7-33e6-4be1-bf98-22c786edc3bb/volumes" Oct 31 13:12:40.026288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:40.026244 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892"] Oct 31 13:12:40.029826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:40.029803 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571bb286-qd892"] Oct 31 13:12:40.099293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:40.099256 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" event={"ID":"e268dce8-438c-4f50-b0ad-7253f98a5eeb","Type":"ContainerStarted","Data":"3d864e068378198937417485491dcbe8c9032484e3e0292e44308376fb1b8a1f"} Oct 31 13:12:40.099440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:40.099346 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" event={"ID":"e268dce8-438c-4f50-b0ad-7253f98a5eeb","Type":"ContainerStarted","Data":"f98f8e64d051c21a4e68318ba5b1eb5aa940f53d064c2aff9fdc36ca8e3c87cd"} Oct 31 13:12:40.102840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:40.102809 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" event={"ID":"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070","Type":"ContainerStarted","Data":"e5a31ec797bb8d5e26fd39c20a863a82b41bb438afecf4942cf3de8042800e85"} Oct 31 13:12:40.131178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:40.131128 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" podStartSLOduration=2.131114076 podStartE2EDuration="2.131114076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:40.127313584 +0000 UTC m=+2454.305031569" watchObservedRunningTime="2026-10-31 13:12:40.131114076 +0000 UTC m=+2454.308832059" Oct 31 13:12:41.114590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.114548 5074 generic.go:356] "Generic (PLEG): container finished" podID="25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" containerID="ec64e76eac77395fdac4bec94db747b869720290196a81473beaa280cdca9328" exitCode=0 Oct 31 13:12:41.115063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.114651 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" event={"ID":"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070","Type":"ContainerDied","Data":"ec64e76eac77395fdac4bec94db747b869720290196a81473beaa280cdca9328"} Oct 31 13:12:41.116707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.116680 5074 generic.go:356] "Generic (PLEG): container finished" podID="e268dce8-438c-4f50-b0ad-7253f98a5eeb" containerID="3d864e068378198937417485491dcbe8c9032484e3e0292e44308376fb1b8a1f" exitCode=0 Oct 31 13:12:41.116840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.116757 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" event={"ID":"e268dce8-438c-4f50-b0ad-7253f98a5eeb","Type":"ContainerDied","Data":"3d864e068378198937417485491dcbe8c9032484e3e0292e44308376fb1b8a1f"} Oct 31 13:12:41.175508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.175470 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr"] Oct 31 13:12:41.180673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.180648 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.195440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.195402 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr"] Oct 31 13:12:41.246135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.245963 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvbv\" (UniqueName: \"kubernetes.io/projected/00af11da-a413-405d-be46-651ae8accff3-kube-api-access-7rvbv\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.246135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.246127 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af11da-a413-405d-be46-651ae8accff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.246330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.246248 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af11da-a413-405d-be46-651ae8accff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.349841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.348071 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af11da-a413-405d-be46-651ae8accff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.349841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.348227 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af11da-a413-405d-be46-651ae8accff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.349841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.348434 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvbv\" (UniqueName: \"kubernetes.io/projected/00af11da-a413-405d-be46-651ae8accff3-kube-api-access-7rvbv\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.350280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.350193 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af11da-a413-405d-be46-651ae8accff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.352120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.352090 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af11da-a413-405d-be46-651ae8accff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.370750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.370674 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvbv\" (UniqueName: \"kubernetes.io/projected/00af11da-a413-405d-be46-651ae8accff3-kube-api-access-7rvbv\") pod \"maas-api-revoke-sub-68946767-7nwbr\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.515045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.515001 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:41.805694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.805591 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr"] Oct 31 13:12:41.940528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:41.940492 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f31606-0ffe-4b24-98ed-b01f2b49d773" path="/var/lib/kubelet/pods/a6f31606-0ffe-4b24-98ed-b01f2b49d773/volumes" Oct 31 13:12:42.126860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.126815 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" event={"ID":"00af11da-a413-405d-be46-651ae8accff3","Type":"ContainerStarted","Data":"24a7912fad650d9a951c45fad59d9353a5357af03b34d116572f65d012e39368"} Oct 31 13:12:42.417653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.413282 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:42.468605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.468575 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:42.477928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.477895 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-openshift-ca\") pod \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " Oct 31 13:12:42.478139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.478115 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-xks-ca\") pod \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " Oct 31 13:12:42.478340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.478322 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-kube-api-access-r6jhj\" (UniqueName: \"kubernetes.io/projected/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-kube-api-access-r6jhj\") pod \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\" (UID: \"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070\") " Oct 31 13:12:42.481818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.481788 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-xks-ca" pod "25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" (UID: "25f4ddd7-ccbb-4858-87c3-f7b69dbe9070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:42.481961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.481845 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-openshift-ca" pod "25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" (UID: "25f4ddd7-ccbb-4858-87c3-f7b69dbe9070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:42.482356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.482331 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-kube-api-access-r6jhj" pod "25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" (UID: "25f4ddd7-ccbb-4858-87c3-f7b69dbe9070"). InnerVolumeSpecName "kube-api-access-r6jhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:42.580435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.580400 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-xks-ca\") pod \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " Oct 31 13:12:42.580696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.580488 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-openshift-ca\") pod \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " Oct 31 13:12:42.580696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.580575 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e268dce8-438c-4f50-b0ad-7253f98a5eeb-kube-api-access-dw6kr\" (UniqueName: \"kubernetes.io/projected/e268dce8-438c-4f50-b0ad-7253f98a5eeb-kube-api-access-dw6kr\") pod \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\" (UID: \"e268dce8-438c-4f50-b0ad-7253f98a5eeb\") " Oct 31 13:12:42.580997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.580971 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-openshift-ca" pod "e268dce8-438c-4f50-b0ad-7253f98a5eeb" (UID: "e268dce8-438c-4f50-b0ad-7253f98a5eeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:42.581547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.581519 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:42.581547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.581549 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:42.581785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.581563 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6jhj\" (UniqueName: \"kubernetes.io/projected/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-kube-api-access-r6jhj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:42.581785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.581577 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:42.583331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.583306 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-xks-ca" pod "e268dce8-438c-4f50-b0ad-7253f98a5eeb" (UID: "e268dce8-438c-4f50-b0ad-7253f98a5eeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:42.583725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.583700 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e268dce8-438c-4f50-b0ad-7253f98a5eeb-kube-api-access-dw6kr" pod "e268dce8-438c-4f50-b0ad-7253f98a5eeb" (UID: "e268dce8-438c-4f50-b0ad-7253f98a5eeb"). InnerVolumeSpecName "kube-api-access-dw6kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:42.683716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.683613 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e268dce8-438c-4f50-b0ad-7253f98a5eeb-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:42.683716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:42.683668 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6kr\" (UniqueName: \"kubernetes.io/projected/e268dce8-438c-4f50-b0ad-7253f98a5eeb-kube-api-access-dw6kr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:43.135826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.135788 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" Oct 31 13:12:43.136316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.135829 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf" event={"ID":"e268dce8-438c-4f50-b0ad-7253f98a5eeb","Type":"ContainerDied","Data":"f98f8e64d051c21a4e68318ba5b1eb5aa940f53d064c2aff9fdc36ca8e3c87cd"} Oct 31 13:12:43.136316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.135874 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98f8e64d051c21a4e68318ba5b1eb5aa940f53d064c2aff9fdc36ca8e3c87cd" Oct 31 13:12:43.137681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.137625 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" Oct 31 13:12:43.137681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.137665 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9" event={"ID":"25f4ddd7-ccbb-4858-87c3-f7b69dbe9070","Type":"ContainerDied","Data":"e5a31ec797bb8d5e26fd39c20a863a82b41bb438afecf4942cf3de8042800e85"} Oct 31 13:12:43.137857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.137711 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a31ec797bb8d5e26fd39c20a863a82b41bb438afecf4942cf3de8042800e85" Oct 31 13:12:43.140142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.140115 5074 generic.go:356] "Generic (PLEG): container finished" podID="00af11da-a413-405d-be46-651ae8accff3" containerID="3d92350de9d6e688a9b0bb6a4870d0ed9aa99937cd363aeb02509839390927b8" exitCode=0 Oct 31 13:12:43.140250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:43.140195 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" event={"ID":"00af11da-a413-405d-be46-651ae8accff3","Type":"ContainerDied","Data":"3d92350de9d6e688a9b0bb6a4870d0ed9aa99937cd363aeb02509839390927b8"} Oct 31 13:12:44.337379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.337352 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:44.418017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.417908 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00af11da-a413-405d-be46-651ae8accff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af11da-a413-405d-be46-651ae8accff3-maas-api-openshift-ca\") pod \"00af11da-a413-405d-be46-651ae8accff3\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " Oct 31 13:12:44.418200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.418113 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00af11da-a413-405d-be46-651ae8accff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af11da-a413-405d-be46-651ae8accff3-maas-api-xks-ca\") pod \"00af11da-a413-405d-be46-651ae8accff3\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " Oct 31 13:12:44.418200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.418146 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00af11da-a413-405d-be46-651ae8accff3-kube-api-access-7rvbv\" (UniqueName: \"kubernetes.io/projected/00af11da-a413-405d-be46-651ae8accff3-kube-api-access-7rvbv\") pod \"00af11da-a413-405d-be46-651ae8accff3\" (UID: \"00af11da-a413-405d-be46-651ae8accff3\") " Oct 31 13:12:44.418805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.418768 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00af11da-a413-405d-be46-651ae8accff3-maas-api-openshift-ca" pod "00af11da-a413-405d-be46-651ae8accff3" (UID: "00af11da-a413-405d-be46-651ae8accff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:44.418997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.418976 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00af11da-a413-405d-be46-651ae8accff3-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:44.421443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.421378 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00af11da-a413-405d-be46-651ae8accff3-kube-api-access-7rvbv" pod "00af11da-a413-405d-be46-651ae8accff3" (UID: "00af11da-a413-405d-be46-651ae8accff3"). InnerVolumeSpecName "kube-api-access-7rvbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:44.421443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.421419 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00af11da-a413-405d-be46-651ae8accff3-maas-api-xks-ca" pod "00af11da-a413-405d-be46-651ae8accff3" (UID: "00af11da-a413-405d-be46-651ae8accff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:44.520409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.520369 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00af11da-a413-405d-be46-651ae8accff3-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:44.520409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:44.520402 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvbv\" (UniqueName: \"kubernetes.io/projected/00af11da-a413-405d-be46-651ae8accff3-kube-api-access-7rvbv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:45.156691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:45.156661 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" Oct 31 13:12:45.156905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:45.156688 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr" event={"ID":"00af11da-a413-405d-be46-651ae8accff3","Type":"ContainerDied","Data":"24a7912fad650d9a951c45fad59d9353a5357af03b34d116572f65d012e39368"} Oct 31 13:12:45.156905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:45.156747 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a7912fad650d9a951c45fad59d9353a5357af03b34d116572f65d012e39368" Oct 31 13:12:48.716789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.716752 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw"] Oct 31 13:12:48.717423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717405 5074 cpu_manager.go:414] "containerMap: removing container" podUID="00af11da-a413-405d-be46-651ae8accff3" containerName="revoke-keys" Oct 31 13:12:48.717423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717424 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00af11da-a413-405d-be46-651ae8accff3" containerName="revoke-keys" Oct 31 13:12:48.717521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717442 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e268dce8-438c-4f50-b0ad-7253f98a5eeb" containerName="revoke-keys" Oct 31 13:12:48.717521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717451 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e268dce8-438c-4f50-b0ad-7253f98a5eeb" containerName="revoke-keys" Oct 31 13:12:48.717521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717511 5074 cpu_manager.go:414] "containerMap: removing container" podUID="25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" containerName="revoke-keys" Oct 31 13:12:48.717521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717518 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" containerName="revoke-keys" Oct 31 13:12:48.717659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717629 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" containerName="revoke-keys" Oct 31 13:12:48.717659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717652 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="00af11da-a413-405d-be46-651ae8accff3" containerName="revoke-keys" Oct 31 13:12:48.717728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.717660 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e268dce8-438c-4f50-b0ad-7253f98a5eeb" containerName="revoke-keys" Oct 31 13:12:48.722772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.722749 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.732410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.732385 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:12:48.732410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.732385 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:12:48.732575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.732441 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:12:48.733791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.733774 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:12:48.739312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.739271 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw"] Oct 31 13:12:48.782095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.782060 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qr5\" (UniqueName: \"kubernetes.io/projected/c87b86da-756e-4713-ac88-2b28764de3b1-kube-api-access-d8qr5\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.782289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.782207 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.782392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.782366 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.884287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.884244 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qr5\" (UniqueName: \"kubernetes.io/projected/c87b86da-756e-4713-ac88-2b28764de3b1-kube-api-access-d8qr5\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.884476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.884364 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.884476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.884447 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.885201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.885168 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.886913 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.886885 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.897172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.897129 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qr5\" (UniqueName: \"kubernetes.io/projected/c87b86da-756e-4713-ac88-2b28764de3b1-kube-api-access-d8qr5\") pod \"maas-api-revoke-sub-942e5f29-7ktlw\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:48.997667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:48.997582 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k"] Oct 31 13:12:49.002222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.002203 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.014788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.014742 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k"] Oct 31 13:12:49.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.038041 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:49.087598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.087564 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9sv\" (UniqueName: \"kubernetes.io/projected/0bcc14a0-2bab-4527-a2d3-3925673b8b45-kube-api-access-kw9sv\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.087775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.087620 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.087775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.087731 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.189417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.189367 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9sv\" (UniqueName: \"kubernetes.io/projected/0bcc14a0-2bab-4527-a2d3-3925673b8b45-kube-api-access-kw9sv\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.189614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.189433 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.189614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.189474 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.190420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.190392 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.196247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.196211 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.215244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.210145 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9sv\" (UniqueName: \"kubernetes.io/projected/0bcc14a0-2bab-4527-a2d3-3925673b8b45-kube-api-access-kw9sv\") pod \"maas-api-revoke-sub-df35a4a8-26s4k\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.320982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.320869 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:49.379822 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:49.379238 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87b86da_756e_4713_ac88_2b28764de3b1.slice/crio-91aeec688a733999bb19fa01f499673f1463bdcd686376701a6551748de7eee7 WatchSource:0}: Error finding container 91aeec688a733999bb19fa01f499673f1463bdcd686376701a6551748de7eee7: Status 404 returned error can't find the container with id 91aeec688a733999bb19fa01f499673f1463bdcd686376701a6551748de7eee7 Oct 31 13:12:49.387922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.387208 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw"] Oct 31 13:12:49.881271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.881229 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k"] Oct 31 13:12:49.881271 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:49.881247 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcc14a0_2bab_4527_a2d3_3925673b8b45.slice/crio-e378a2609b55cb10d7c5eff8759127fc4b0917156fee35bbe6a58b858c11d27a WatchSource:0}: Error finding container e378a2609b55cb10d7c5eff8759127fc4b0917156fee35bbe6a58b858c11d27a: Status 404 returned error can't find the container with id e378a2609b55cb10d7c5eff8759127fc4b0917156fee35bbe6a58b858c11d27a Oct 31 13:12:49.968216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.968182 5074 scope.go:122] "RemoveContainer" containerID="8f417946c781b3312881959576a6675c9a62dd93ed9e908811125765f8117e3b" Oct 31 13:12:49.998438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:49.998416 5074 scope.go:122] "RemoveContainer" containerID="071395dc6c238429824d66ba4b1262fce7eaeb9778e8933a36a3221350a775fb" Oct 31 13:12:50.008727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.008689 5074 scope.go:122] "RemoveContainer" containerID="59c655c487c9cb271cd4260b3b9b4828a4aa3d13d401b7e13fc8a4af7766bb9d" Oct 31 13:12:50.018932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.018903 5074 scope.go:122] "RemoveContainer" containerID="ceb20ecd49ce98013f4864867a48509b8def8af4a94712f82d3c76e8e6c193f4" Oct 31 13:12:50.028492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.028472 5074 scope.go:122] "RemoveContainer" containerID="5ffdf902cac1aa033a2aadd7950c58c357195fb299f96a5812d95b8c468a6f71" Oct 31 13:12:50.037350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.037330 5074 scope.go:122] "RemoveContainer" containerID="968226a9e28e620d3dfa78d2637000790061d56c54a927c0c2d5745bc9fe36cb" Oct 31 13:12:50.048877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.048707 5074 scope.go:122] "RemoveContainer" containerID="697c283022db83f1f4e562546467f145409c4682510cd7a8d350633f98f9b889" Oct 31 13:12:50.210157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.210120 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" event={"ID":"0bcc14a0-2bab-4527-a2d3-3925673b8b45","Type":"ContainerStarted","Data":"e378a2609b55cb10d7c5eff8759127fc4b0917156fee35bbe6a58b858c11d27a"} Oct 31 13:12:50.213366 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.213340 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" event={"ID":"c87b86da-756e-4713-ac88-2b28764de3b1","Type":"ContainerStarted","Data":"e61ba8a1c9e21f1ef78304ab830991fdb081b596b886148ac89a5ca7c8e65780"} Oct 31 13:12:50.213505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.213390 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" event={"ID":"c87b86da-756e-4713-ac88-2b28764de3b1","Type":"ContainerStarted","Data":"91aeec688a733999bb19fa01f499673f1463bdcd686376701a6551748de7eee7"} Oct 31 13:12:50.238589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:50.238536 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" podStartSLOduration=2.238521992 podStartE2EDuration="2.238521992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:50.233662046 +0000 UTC m=+2464.411380031" watchObservedRunningTime="2026-10-31 13:12:50.238521992 +0000 UTC m=+2464.416240018" Oct 31 13:12:51.228357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.228326 5074 generic.go:356] "Generic (PLEG): container finished" podID="0bcc14a0-2bab-4527-a2d3-3925673b8b45" containerID="dfc83079c05362b7e6d6578b1414e0ee37f105494b41694227632cba599c9857" exitCode=0 Oct 31 13:12:51.228833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.228398 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" event={"ID":"0bcc14a0-2bab-4527-a2d3-3925673b8b45","Type":"ContainerDied","Data":"dfc83079c05362b7e6d6578b1414e0ee37f105494b41694227632cba599c9857"} Oct 31 13:12:51.230124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.230099 5074 generic.go:356] "Generic (PLEG): container finished" podID="c87b86da-756e-4713-ac88-2b28764de3b1" containerID="e61ba8a1c9e21f1ef78304ab830991fdb081b596b886148ac89a5ca7c8e65780" exitCode=0 Oct 31 13:12:51.230243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.230183 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" event={"ID":"c87b86da-756e-4713-ac88-2b28764de3b1","Type":"ContainerDied","Data":"e61ba8a1c9e21f1ef78304ab830991fdb081b596b886148ac89a5ca7c8e65780"} Oct 31 13:12:51.459872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.459777 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts"] Oct 31 13:12:51.469862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.467460 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.469862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.469447 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts"] Oct 31 13:12:51.552280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.552238 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfhg\" (UniqueName: \"kubernetes.io/projected/c9851471-2652-4d93-a001-0288b3905e40-kube-api-access-6pfhg\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.552478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.552417 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9851471-2652-4d93-a001-0288b3905e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.552680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.552627 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9851471-2652-4d93-a001-0288b3905e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.654547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.654512 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9851471-2652-4d93-a001-0288b3905e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.654741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.654599 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfhg\" (UniqueName: \"kubernetes.io/projected/c9851471-2652-4d93-a001-0288b3905e40-kube-api-access-6pfhg\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.654741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.654738 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9851471-2652-4d93-a001-0288b3905e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.655387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.655360 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9851471-2652-4d93-a001-0288b3905e40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.656893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.656868 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9851471-2652-4d93-a001-0288b3905e40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.666648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.666615 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfhg\" (UniqueName: \"kubernetes.io/projected/c9851471-2652-4d93-a001-0288b3905e40-kube-api-access-6pfhg\") pod \"maas-api-revoke-sub-76aa74be-wg9ts\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:51.816057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:51.816023 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:52.127932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.127836 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts"] Oct 31 13:12:52.134928 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:52.134848 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9851471_2652_4d93_a001_0288b3905e40.slice/crio-a12482d6fbaa5f4d21b85cbd7555e9814ddcb7752fae8b52e424062dbe217048 WatchSource:0}: Error finding container a12482d6fbaa5f4d21b85cbd7555e9814ddcb7752fae8b52e424062dbe217048: Status 404 returned error can't find the container with id a12482d6fbaa5f4d21b85cbd7555e9814ddcb7752fae8b52e424062dbe217048 Oct 31 13:12:52.239171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.239128 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" event={"ID":"c9851471-2652-4d93-a001-0288b3905e40","Type":"ContainerStarted","Data":"a12482d6fbaa5f4d21b85cbd7555e9814ddcb7752fae8b52e424062dbe217048"} Oct 31 13:12:52.462565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.462531 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:52.503152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.503126 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:52.574917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.574882 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87b86da-756e-4713-ac88-2b28764de3b1-kube-api-access-d8qr5\" (UniqueName: \"kubernetes.io/projected/c87b86da-756e-4713-ac88-2b28764de3b1-kube-api-access-d8qr5\") pod \"c87b86da-756e-4713-ac88-2b28764de3b1\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " Oct 31 13:12:52.575110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.574998 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcc14a0-2bab-4527-a2d3-3925673b8b45-kube-api-access-kw9sv\" (UniqueName: \"kubernetes.io/projected/0bcc14a0-2bab-4527-a2d3-3925673b8b45-kube-api-access-kw9sv\") pod \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " Oct 31 13:12:52.575110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.575089 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-xks-ca\") pod \"c87b86da-756e-4713-ac88-2b28764de3b1\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " Oct 31 13:12:52.575358 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.575339 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-openshift-ca\") pod \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " Oct 31 13:12:52.575439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.575419 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-xks-ca\") pod \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\" (UID: \"0bcc14a0-2bab-4527-a2d3-3925673b8b45\") " Oct 31 13:12:52.575516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.575453 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-openshift-ca\") pod \"c87b86da-756e-4713-ac88-2b28764de3b1\" (UID: \"c87b86da-756e-4713-ac88-2b28764de3b1\") " Oct 31 13:12:52.576271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.576145 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-openshift-ca" pod "0bcc14a0-2bab-4527-a2d3-3925673b8b45" (UID: "0bcc14a0-2bab-4527-a2d3-3925673b8b45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:52.576448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.576382 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-openshift-ca" pod "c87b86da-756e-4713-ac88-2b28764de3b1" (UID: "c87b86da-756e-4713-ac88-2b28764de3b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:52.577508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.577466 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87b86da-756e-4713-ac88-2b28764de3b1-kube-api-access-d8qr5" pod "c87b86da-756e-4713-ac88-2b28764de3b1" (UID: "c87b86da-756e-4713-ac88-2b28764de3b1"). InnerVolumeSpecName "kube-api-access-d8qr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:52.577617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.577592 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcc14a0-2bab-4527-a2d3-3925673b8b45-kube-api-access-kw9sv" pod "0bcc14a0-2bab-4527-a2d3-3925673b8b45" (UID: "0bcc14a0-2bab-4527-a2d3-3925673b8b45"). InnerVolumeSpecName "kube-api-access-kw9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:52.578658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.578616 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-xks-ca" pod "c87b86da-756e-4713-ac88-2b28764de3b1" (UID: "c87b86da-756e-4713-ac88-2b28764de3b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:52.578748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.578726 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-xks-ca" pod "0bcc14a0-2bab-4527-a2d3-3925673b8b45" (UID: "0bcc14a0-2bab-4527-a2d3-3925673b8b45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:52.677363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.677265 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:52.677363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.677297 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcc14a0-2bab-4527-a2d3-3925673b8b45-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:52.677363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.677307 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:52.677363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.677316 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qr5\" (UniqueName: \"kubernetes.io/projected/c87b86da-756e-4713-ac88-2b28764de3b1-kube-api-access-d8qr5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:52.677363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.677327 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw9sv\" (UniqueName: \"kubernetes.io/projected/0bcc14a0-2bab-4527-a2d3-3925673b8b45-kube-api-access-kw9sv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:52.677363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:52.677335 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c87b86da-756e-4713-ac88-2b28764de3b1-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:53.249382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.249349 5074 generic.go:356] "Generic (PLEG): container finished" podID="c9851471-2652-4d93-a001-0288b3905e40" containerID="e3db449cac2291f70952d965b9ebf3149cbfc33bc56fec70089b5f9007532efc" exitCode=0 Oct 31 13:12:53.249822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.249443 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" event={"ID":"c9851471-2652-4d93-a001-0288b3905e40","Type":"ContainerDied","Data":"e3db449cac2291f70952d965b9ebf3149cbfc33bc56fec70089b5f9007532efc"} Oct 31 13:12:53.251197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.251176 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" Oct 31 13:12:53.251333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.251176 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k" event={"ID":"0bcc14a0-2bab-4527-a2d3-3925673b8b45","Type":"ContainerDied","Data":"e378a2609b55cb10d7c5eff8759127fc4b0917156fee35bbe6a58b858c11d27a"} Oct 31 13:12:53.251333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.251292 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e378a2609b55cb10d7c5eff8759127fc4b0917156fee35bbe6a58b858c11d27a" Oct 31 13:12:53.252849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.252830 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" Oct 31 13:12:53.252946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.252865 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw" event={"ID":"c87b86da-756e-4713-ac88-2b28764de3b1","Type":"ContainerDied","Data":"91aeec688a733999bb19fa01f499673f1463bdcd686376701a6551748de7eee7"} Oct 31 13:12:53.252946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:53.252912 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91aeec688a733999bb19fa01f499673f1463bdcd686376701a6551748de7eee7" Oct 31 13:12:54.455287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.455258 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:54.529058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.528967 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9851471-2652-4d93-a001-0288b3905e40-kube-api-access-6pfhg\" (UniqueName: \"kubernetes.io/projected/c9851471-2652-4d93-a001-0288b3905e40-kube-api-access-6pfhg\") pod \"c9851471-2652-4d93-a001-0288b3905e40\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " Oct 31 13:12:54.529265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.529238 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9851471-2652-4d93-a001-0288b3905e40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9851471-2652-4d93-a001-0288b3905e40-maas-api-openshift-ca\") pod \"c9851471-2652-4d93-a001-0288b3905e40\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " Oct 31 13:12:54.529359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.529332 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9851471-2652-4d93-a001-0288b3905e40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9851471-2652-4d93-a001-0288b3905e40-maas-api-xks-ca\") pod \"c9851471-2652-4d93-a001-0288b3905e40\" (UID: \"c9851471-2652-4d93-a001-0288b3905e40\") " Oct 31 13:12:54.529778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.529744 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9851471-2652-4d93-a001-0288b3905e40-maas-api-openshift-ca" pod "c9851471-2652-4d93-a001-0288b3905e40" (UID: "c9851471-2652-4d93-a001-0288b3905e40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:12:54.530161 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.530133 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9851471-2652-4d93-a001-0288b3905e40-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:54.531294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.531268 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9851471-2652-4d93-a001-0288b3905e40-kube-api-access-6pfhg" pod "c9851471-2652-4d93-a001-0288b3905e40" (UID: "c9851471-2652-4d93-a001-0288b3905e40"). InnerVolumeSpecName "kube-api-access-6pfhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:12:54.531573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.531557 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9851471-2652-4d93-a001-0288b3905e40-maas-api-xks-ca" pod "c9851471-2652-4d93-a001-0288b3905e40" (UID: "c9851471-2652-4d93-a001-0288b3905e40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:12:54.631537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.631499 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9851471-2652-4d93-a001-0288b3905e40-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:54.631537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.631531 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfhg\" (UniqueName: \"kubernetes.io/projected/c9851471-2652-4d93-a001-0288b3905e40-kube-api-access-6pfhg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:12:54.952575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.952465 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2"] Oct 31 13:12:54.953445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953427 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c9851471-2652-4d93-a001-0288b3905e40" containerName="revoke-keys" Oct 31 13:12:54.953502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953448 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9851471-2652-4d93-a001-0288b3905e40" containerName="revoke-keys" Oct 31 13:12:54.953502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953474 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0bcc14a0-2bab-4527-a2d3-3925673b8b45" containerName="revoke-keys" Oct 31 13:12:54.953502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953483 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcc14a0-2bab-4527-a2d3-3925673b8b45" containerName="revoke-keys" Oct 31 13:12:54.953605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953525 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c87b86da-756e-4713-ac88-2b28764de3b1" containerName="revoke-keys" Oct 31 13:12:54.953605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953533 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87b86da-756e-4713-ac88-2b28764de3b1" containerName="revoke-keys" Oct 31 13:12:54.953712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953701 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9851471-2652-4d93-a001-0288b3905e40" containerName="revoke-keys" Oct 31 13:12:54.953749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953722 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcc14a0-2bab-4527-a2d3-3925673b8b45" containerName="revoke-keys" Oct 31 13:12:54.953749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.953738 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87b86da-756e-4713-ac88-2b28764de3b1" containerName="revoke-keys" Oct 31 13:12:54.958012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.957992 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:54.969232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.969204 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-15a6a2\"" Oct 31 13:12:54.969828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.969811 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-15a6a2-dockercfg-x84qj\"" Oct 31 13:12:54.971061 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.971039 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2"] Oct 31 13:12:54.979817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.979776 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9"] Oct 31 13:12:54.985588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:54.985504 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.008274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.008246 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9"] Oct 31 13:12:55.044943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.044841 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2wh\" (UniqueName: \"kubernetes.io/projected/38f511f1-b31c-44aa-a6de-04741bfe6f08-kube-api-access-nw2wh\") pod \"payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:55.045480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.045446 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f511f1-b31c-44aa-a6de-04741bfe6f08-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:55.223713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.189046 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f511f1-b31c-44aa-a6de-04741bfe6f08-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:55.311342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.310814 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgb62\" (UniqueName: \"kubernetes.io/projected/5529ab24-7af7-4584-94cf-c6a6f485cd3f-kube-api-access-cgb62\") pod \"payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.311342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.194796 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f511f1-b31c-44aa-a6de-04741bfe6f08-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:55.311342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.311042 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2wh\" (UniqueName: \"kubernetes.io/projected/38f511f1-b31c-44aa-a6de-04741bfe6f08-kube-api-access-nw2wh\") pod \"payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:55.311342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.311102 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529ab24-7af7-4584-94cf-c6a6f485cd3f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.318877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.318842 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" event={"ID":"c9851471-2652-4d93-a001-0288b3905e40","Type":"ContainerDied","Data":"a12482d6fbaa5f4d21b85cbd7555e9814ddcb7752fae8b52e424062dbe217048"} Oct 31 13:12:55.319035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.318900 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a12482d6fbaa5f4d21b85cbd7555e9814ddcb7752fae8b52e424062dbe217048" Oct 31 13:12:55.319035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.318965 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts" Oct 31 13:12:55.351938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.351912 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2wh\" (UniqueName: \"kubernetes.io/projected/38f511f1-b31c-44aa-a6de-04741bfe6f08-kube-api-access-nw2wh\") pod \"payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:55.412206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.412174 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgb62\" (UniqueName: \"kubernetes.io/projected/5529ab24-7af7-4584-94cf-c6a6f485cd3f-kube-api-access-cgb62\") pod \"payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.412398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.412300 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529ab24-7af7-4584-94cf-c6a6f485cd3f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.413072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.413053 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529ab24-7af7-4584-94cf-c6a6f485cd3f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.429421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.429390 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgb62\" (UniqueName: \"kubernetes.io/projected/5529ab24-7af7-4584-94cf-c6a6f485cd3f-kube-api-access-cgb62\") pod \"payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.575548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.575445 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:55.614658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.608002 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:55.884147 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:55.884122 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f511f1_b31c_44aa_a6de_04741bfe6f08.slice/crio-6b692b4b505b0c51503fea65b7b9f3c84dfedf0024b3827eaf3ebe9eb51e7b75 WatchSource:0}: Error finding container 6b692b4b505b0c51503fea65b7b9f3c84dfedf0024b3827eaf3ebe9eb51e7b75: Status 404 returned error can't find the container with id 6b692b4b505b0c51503fea65b7b9f3c84dfedf0024b3827eaf3ebe9eb51e7b75 Oct 31 13:12:55.887486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.887456 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2"] Oct 31 13:12:55.945149 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:55.945124 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9"] Oct 31 13:12:55.945450 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:12:55.945416 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5529ab24_7af7_4584_94cf_c6a6f485cd3f.slice/crio-eb2505aadde424c13a9b92855a8f099be6ae6b741b6b48b769c48c46bae34931 WatchSource:0}: Error finding container eb2505aadde424c13a9b92855a8f099be6ae6b741b6b48b769c48c46bae34931: Status 404 returned error can't find the container with id eb2505aadde424c13a9b92855a8f099be6ae6b741b6b48b769c48c46bae34931 Oct 31 13:12:55.989807 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:12:55.989132 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:12:56.328769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.328727 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" event={"ID":"5529ab24-7af7-4584-94cf-c6a6f485cd3f","Type":"ContainerStarted","Data":"ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a"} Oct 31 13:12:56.328969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.328795 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" event={"ID":"5529ab24-7af7-4584-94cf-c6a6f485cd3f","Type":"ContainerStarted","Data":"eb2505aadde424c13a9b92855a8f099be6ae6b741b6b48b769c48c46bae34931"} Oct 31 13:12:56.328969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.328824 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:12:56.330645 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.330603 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" event={"ID":"38f511f1-b31c-44aa-a6de-04741bfe6f08","Type":"ContainerStarted","Data":"e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82"} Oct 31 13:12:56.330785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.330668 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:12:56.330785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.330689 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" event={"ID":"38f511f1-b31c-44aa-a6de-04741bfe6f08","Type":"ContainerStarted","Data":"6b692b4b505b0c51503fea65b7b9f3c84dfedf0024b3827eaf3ebe9eb51e7b75"} Oct 31 13:12:56.354750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.354700 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" podStartSLOduration=2.354687637 podStartE2EDuration="2.354687637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:56.353328163 +0000 UTC m=+2470.531046147" watchObservedRunningTime="2026-10-31 13:12:56.354687637 +0000 UTC m=+2470.532405620" Oct 31 13:12:56.392362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:12:56.392305 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" podStartSLOduration=2.392288823 podStartE2EDuration="2.392288823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:56.391177566 +0000 UTC m=+2470.568895553" watchObservedRunningTime="2026-10-31 13:12:56.392288823 +0000 UTC m=+2470.570006809" Oct 31 13:13:00.714541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:00.711380 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6"] Oct 31 13:13:07.345821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:07.345226 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:13:07.373265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:07.372702 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:13:11.596475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.595613 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k"] Oct 31 13:13:11.630558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.623983 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp"] Oct 31 13:13:11.630558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.624118 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.635799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.635772 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k"] Oct 31 13:13:11.636104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.636042 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:11.640289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.640260 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp"] Oct 31 13:13:11.644117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.644076 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-54a16b\"" Oct 31 13:13:11.644294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.644280 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-54a16b-dockercfg-hk2v4\"" Oct 31 13:13:11.654766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.654743 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82q6\" (UniqueName: \"kubernetes.io/projected/cc8a91e2-db37-4692-b1c2-de0c1909e825-kube-api-access-j82q6\") pod \"payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:11.654973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.654955 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2nl6\" (UniqueName: \"kubernetes.io/projected/faa58097-a14b-4f50-81fe-b2d3265f4d29-kube-api-access-r2nl6\") pod \"payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.655302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.655285 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a91e2-db37-4692-b1c2-de0c1909e825-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:11.655474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.655460 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa58097-a14b-4f50-81fe-b2d3265f4d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.785658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.778311 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82q6\" (UniqueName: \"kubernetes.io/projected/cc8a91e2-db37-4692-b1c2-de0c1909e825-kube-api-access-j82q6\") pod \"payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:11.786378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.786346 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2nl6\" (UniqueName: \"kubernetes.io/projected/faa58097-a14b-4f50-81fe-b2d3265f4d29-kube-api-access-r2nl6\") pod \"payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.787068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.787046 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a91e2-db37-4692-b1c2-de0c1909e825-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:11.813901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.813869 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa58097-a14b-4f50-81fe-b2d3265f4d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.815099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.813674 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a91e2-db37-4692-b1c2-de0c1909e825-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:11.815264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.814891 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa58097-a14b-4f50-81fe-b2d3265f4d29-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.829361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.820907 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82q6\" (UniqueName: \"kubernetes.io/projected/cc8a91e2-db37-4692-b1c2-de0c1909e825-kube-api-access-j82q6\") pod \"payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:11.863705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.861514 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2nl6\" (UniqueName: \"kubernetes.io/projected/faa58097-a14b-4f50-81fe-b2d3265f4d29-kube-api-access-r2nl6\") pod \"payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.967656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.960870 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:11.974022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:11.973419 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:12.743007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:12.742966 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k"] Oct 31 13:13:12.857177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:12.855786 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp"] Oct 31 13:13:13.700657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.700524 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" event={"ID":"faa58097-a14b-4f50-81fe-b2d3265f4d29","Type":"ContainerStarted","Data":"3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef"} Oct 31 13:13:13.700657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.700592 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" event={"ID":"faa58097-a14b-4f50-81fe-b2d3265f4d29","Type":"ContainerStarted","Data":"5141df6fcfe823fcfaa8e7e1b70282dbc1872e64884d6d513b5e2a46a3e7ba47"} Oct 31 13:13:13.718656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.713734 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:13.718656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.716839 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" event={"ID":"cc8a91e2-db37-4692-b1c2-de0c1909e825","Type":"ContainerStarted","Data":"5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238"} Oct 31 13:13:13.718656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.716901 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" event={"ID":"cc8a91e2-db37-4692-b1c2-de0c1909e825","Type":"ContainerStarted","Data":"718d64857c2db07370406ca46b0c2ae4b6d3499ff36b3d4d155534da72f26ad2"} Oct 31 13:13:13.718656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.717894 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:13.754657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.754031 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" podStartSLOduration=2.754008149 podStartE2EDuration="2.754008149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:13.75130326 +0000 UTC m=+2487.929021245" watchObservedRunningTime="2026-10-31 13:13:13.754008149 +0000 UTC m=+2487.931726132" Oct 31 13:13:13.831819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:13.831749 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" podStartSLOduration=2.831729423 podStartE2EDuration="2.831729423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:13.783647337 +0000 UTC m=+2487.961365319" watchObservedRunningTime="2026-10-31 13:13:13.831729423 +0000 UTC m=+2488.009447408" Oct 31 13:13:15.743000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:15.742944 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" podUID="ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" containerName="main" containerID="cri-o://3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2" gracePeriod=45 Oct 31 13:13:16.118890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.118833 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2"] Oct 31 13:13:16.332882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.332847 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:13:16.628710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.613520 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-home\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-home\") pod \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " Oct 31 13:13:16.628710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.613683 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tls-certs\") pod \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " Oct 31 13:13:16.628710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.613781 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tmp-dir\") pod \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " Oct 31 13:13:16.628710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.613858 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-kube-api-access-wfj9d\" (UniqueName: \"kubernetes.io/projected/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-kube-api-access-wfj9d\") pod \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " Oct 31 13:13:16.628710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.613895 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-dshm\") pod \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " Oct 31 13:13:16.628710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.614079 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-model-cache\") pod \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\" (UID: \"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b\") " Oct 31 13:13:16.628710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.616918 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-home" pod "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" (UID: "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:16.629229 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.628740 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tmp-dir" pod "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" (UID: "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:16.651141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.651105 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-model-cache" pod "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" (UID: "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:16.657435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.657378 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tls-certs" pod "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" (UID: "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:16.658313 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.658259 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-dshm" pod "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" (UID: "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:16.659115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.659086 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-kube-api-access-wfj9d" pod "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" (UID: "ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b"). InnerVolumeSpecName "kube-api-access-wfj9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:16.718659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.717805 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:16.718659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.717848 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:16.718659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.717863 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:16.718659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.717879 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfj9d\" (UniqueName: \"kubernetes.io/projected/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-kube-api-access-wfj9d\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:16.718659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.717895 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:16.718659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.717911 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:16.827347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.827311 5074 generic.go:356] "Generic (PLEG): container finished" podID="ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" containerID="3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2" exitCode=0 Oct 31 13:13:16.827846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.827360 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" event={"ID":"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b","Type":"ContainerDied","Data":"3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2"} Oct 31 13:13:16.827846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.827369 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" Oct 31 13:13:16.827846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.827391 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" event={"ID":"ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b","Type":"ContainerDied","Data":"e9b3d6dc4af0b99bc946c60a3dee216426d7f6a1ba3b068bbcd0c47b26af90aa"} Oct 31 13:13:16.827846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.827419 5074 scope.go:122] "RemoveContainer" containerID="3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2" Oct 31 13:13:16.840095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.840073 5074 scope.go:122] "RemoveContainer" containerID="3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2" Oct 31 13:13:16.840421 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:16.840392 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2\": container with ID starting with 3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2 not found: ID does not exist" containerID="3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2" Oct 31 13:13:16.840466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.840434 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2"} err="failed to get container status \"3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2\": rpc error: code = NotFound desc = could not find container \"3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2\": container with ID starting with 3fe80150016493727722492f7b0aba2056b3443d081d20a0782f000af3a4dcd2 not found: ID does not exist" Oct 31 13:13:16.862473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.862438 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6"] Oct 31 13:13:16.901095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:16.900998 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6"] Oct 31 13:13:17.062767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.062725 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-45bcad/auth-test-model-45bcad-kserve-5ddddf784f-2gtw6" podUID="ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.187:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:13:17.335432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.335391 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7"] Oct 31 13:13:17.336345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.336315 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" containerName="main" Oct 31 13:13:17.336345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.336342 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" containerName="main" Oct 31 13:13:17.336575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.336559 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" containerName="main" Oct 31 13:13:17.342046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.342021 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.362207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.361460 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-fa8c3f-openshift-default-dockercfg-7qr58\"" Oct 31 13:13:17.427655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.427398 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7"] Oct 31 13:13:17.443126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443088 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pldwq\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-kube-api-access-pldwq\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443143 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-credential-socket\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443181 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-certs\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443253 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-data\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443307 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316711a0-4531-404b-9e48-8c7824f93978-istiod-ca-cert\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443344 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-envoy\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443367 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-istio-token\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443544 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316711a0-4531-404b-9e48-8c7824f93978-istio-podinfo\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.443615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.443601 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-socket\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.546659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.546553 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pldwq\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-kube-api-access-pldwq\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.546888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.546623 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-credential-socket\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.546888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.546723 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-certs\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.546888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.546823 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-data\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.547066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.546893 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316711a0-4531-404b-9e48-8c7824f93978-istiod-ca-cert\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.547066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.546936 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-envoy\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.547066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.546959 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-istio-token\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.547243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.547194 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316711a0-4531-404b-9e48-8c7824f93978-istio-podinfo\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.547305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.547256 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-socket\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.596730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.595865 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-socket\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.596903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.596818 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-certs\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.676841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.676509 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316711a0-4531-404b-9e48-8c7824f93978-istio-podinfo\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.677876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.677626 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pldwq\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-kube-api-access-pldwq\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.678870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.678822 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-istio-token\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.715658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.715298 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316711a0-4531-404b-9e48-8c7824f93978-istiod-ca-cert\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.723714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.716503 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-data\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.723714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.716724 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-credential-socket\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.729710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.728048 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-envoy\") pod \"e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:17.986780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:17.970316 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:18.130355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.035031 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b" path="/var/lib/kubelet/pods/ffcfbe11-91d1-4ec9-8ee6-697bfc85d10b/volumes" Oct 31 13:13:18.157770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.157735 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2"] Oct 31 13:13:18.165780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.164738 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:18.254684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.254597 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5dc570-dockercfg-ld59t\"" Oct 31 13:13:18.294232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.273674 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5dc570\"" Oct 31 13:13:18.294464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.280783 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2348a7b1-ee73-4386-8f9d-060c1f23aed6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:18.324256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.324217 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4slhg\" (UniqueName: \"kubernetes.io/projected/2348a7b1-ee73-4386-8f9d-060c1f23aed6-kube-api-access-4slhg\") pod \"payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:18.421657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.421152 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2"] Oct 31 13:13:18.468665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.436560 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4slhg\" (UniqueName: \"kubernetes.io/projected/2348a7b1-ee73-4386-8f9d-060c1f23aed6-kube-api-access-4slhg\") pod \"payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:18.532714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.522846 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2348a7b1-ee73-4386-8f9d-060c1f23aed6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:18.568729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.551367 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2348a7b1-ee73-4386-8f9d-060c1f23aed6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:18.678741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.678693 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4slhg\" (UniqueName: \"kubernetes.io/projected/2348a7b1-ee73-4386-8f9d-060c1f23aed6-kube-api-access-4slhg\") pod \"payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:18.899080 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:18.898239 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:19.027171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:19.024774 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7"] Oct 31 13:13:19.042031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:19.041600 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:19.042031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:19.041725 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:19.042031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:19.041773 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:19.819234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:19.819187 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2"] Oct 31 13:13:20.032698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:20.032312 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:20.032698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:20.032377 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" event={"ID":"316711a0-4531-404b-9e48-8c7824f93978","Type":"ContainerStarted","Data":"e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269"} Oct 31 13:13:20.032698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:20.032410 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" event={"ID":"316711a0-4531-404b-9e48-8c7824f93978","Type":"ContainerStarted","Data":"9cb871cb133eece43dcad890967510fa9442516afa258a8d7d5e2a43d5291762"} Oct 31 13:13:20.043736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:20.038087 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" event={"ID":"2348a7b1-ee73-4386-8f9d-060c1f23aed6","Type":"ContainerStarted","Data":"42b512b024dc58b721fc723fdf972d4707c045d4e9b745c4e7ddefdfcc26bf4d"} Oct 31 13:13:20.054239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:20.054166 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podStartSLOduration=3.054145874 podStartE2EDuration="3.054145874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:20.046928683 +0000 UTC m=+2494.224646668" watchObservedRunningTime="2026-10-31 13:13:20.054145874 +0000 UTC m=+2494.231863859" Oct 31 13:13:21.004352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:21.004017 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:21.209656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:21.186300 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" event={"ID":"2348a7b1-ee73-4386-8f9d-060c1f23aed6","Type":"ContainerStarted","Data":"5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9"} Oct 31 13:13:21.209656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:21.186375 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:21.209656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:21.186438 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:21.213283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:21.210718 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:13:21.240053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:21.239942 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" podStartSLOduration=3.23992417 podStartE2EDuration="3.23992417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:21.239123578 +0000 UTC m=+2495.416841564" watchObservedRunningTime="2026-10-31 13:13:21.23992417 +0000 UTC m=+2495.417642156" Oct 31 13:13:25.823052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:25.823021 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:13:25.823479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:25.823106 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:13:26.075567 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:26.075480 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:13:26.230534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:26.230499 5074 generic.go:356] "Generic (PLEG): container finished" podID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerID="1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157" exitCode=0 Oct 31 13:13:26.230738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:26.230572 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" event={"ID":"127032f3-bc89-42ff-ae41-c1ff7e9b0edb","Type":"ContainerDied","Data":"1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157"} Oct 31 13:13:26.231152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:26.231133 5074 scope.go:122] "RemoveContainer" containerID="1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157" Oct 31 13:13:26.233167 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:26.233132 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:26.233264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:26.233225 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:26.233320 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:26.233263 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:27.241508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:27.241467 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" event={"ID":"127032f3-bc89-42ff-ae41-c1ff7e9b0edb","Type":"ContainerStarted","Data":"84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f"} Oct 31 13:13:27.946805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:27.946773 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:13:27.947030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:27.947003 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:13:27.947113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:27.947073 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 13:13:28.958179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:28.952056 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:13:28.958179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:28.952120 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 13:13:29.946905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:29.946873 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:13:29.947096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:29.946931 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 13:13:30.826744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:30.826700 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr"] Oct 31 13:13:30.827452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:30.826949 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" containerID="cri-o://4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41" gracePeriod=30 Oct 31 13:13:30.949429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:30.949389 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:13:30.950540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:30.949996 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 13:13:31.254157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.254097 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" podUID="0c2e4a60-039e-4fef-af10-0ece9707bd61" containerName="main" containerID="cri-o://7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40" gracePeriod=45 Oct 31 13:13:31.441456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.441116 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" podUID="0c2e4a60-039e-4fef-af10-0ece9707bd61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.189:8000/ready\": dial tcp 10.132.0.189:8000: connect: connection refused" Oct 31 13:13:31.738725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.738698 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.800137 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-model-cache\") pod \"0c2e4a60-039e-4fef-af10-0ece9707bd61\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.800209 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2e4a60-039e-4fef-af10-0ece9707bd61-kube-api-access-tnzwz\" (UniqueName: \"kubernetes.io/projected/0c2e4a60-039e-4fef-af10-0ece9707bd61-kube-api-access-tnzwz\") pod \"0c2e4a60-039e-4fef-af10-0ece9707bd61\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.800440 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-tmp-dir\") pod \"0c2e4a60-039e-4fef-af10-0ece9707bd61\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.800558 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c2e4a60-039e-4fef-af10-0ece9707bd61-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2e4a60-039e-4fef-af10-0ece9707bd61-tls-certs\") pod \"0c2e4a60-039e-4fef-af10-0ece9707bd61\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.800675 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-dshm\") pod \"0c2e4a60-039e-4fef-af10-0ece9707bd61\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.800742 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-home\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-home\") pod \"0c2e4a60-039e-4fef-af10-0ece9707bd61\" (UID: \"0c2e4a60-039e-4fef-af10-0ece9707bd61\") " Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.800760 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-model-cache" pod "0c2e4a60-039e-4fef-af10-0ece9707bd61" (UID: "0c2e4a60-039e-4fef-af10-0ece9707bd61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.801483 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:31.801782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.801732 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-home" pod "0c2e4a60-039e-4fef-af10-0ece9707bd61" (UID: "0c2e4a60-039e-4fef-af10-0ece9707bd61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:31.802768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.802522 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-tmp-dir" pod "0c2e4a60-039e-4fef-af10-0ece9707bd61" (UID: "0c2e4a60-039e-4fef-af10-0ece9707bd61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:31.804290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.804255 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2e4a60-039e-4fef-af10-0ece9707bd61-kube-api-access-tnzwz" pod "0c2e4a60-039e-4fef-af10-0ece9707bd61" (UID: "0c2e4a60-039e-4fef-af10-0ece9707bd61"). InnerVolumeSpecName "kube-api-access-tnzwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:31.806406 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.805041 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2e4a60-039e-4fef-af10-0ece9707bd61-tls-certs" pod "0c2e4a60-039e-4fef-af10-0ece9707bd61" (UID: "0c2e4a60-039e-4fef-af10-0ece9707bd61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:13:31.807566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.807540 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-dshm" pod "0c2e4a60-039e-4fef-af10-0ece9707bd61" (UID: "0c2e4a60-039e-4fef-af10-0ece9707bd61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:31.906242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.906154 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:31.907078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.907054 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2e4a60-039e-4fef-af10-0ece9707bd61-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:31.907269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.907239 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:31.907403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.907389 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2e4a60-039e-4fef-af10-0ece9707bd61-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:31.907542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.907528 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnzwz\" (UniqueName: \"kubernetes.io/projected/0c2e4a60-039e-4fef-af10-0ece9707bd61-kube-api-access-tnzwz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:31.946716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.946679 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 13:13:31.946890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:31.946745 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 13:13:32.009887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.009848 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn"] Oct 31 13:13:32.010116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.010088 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" containerID="cri-o://ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae" gracePeriod=30 Oct 31 13:13:32.402102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.402054 5074 generic.go:356] "Generic (PLEG): container finished" podID="0c2e4a60-039e-4fef-af10-0ece9707bd61" containerID="7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40" exitCode=0 Oct 31 13:13:32.402497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.402470 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" event={"ID":"0c2e4a60-039e-4fef-af10-0ece9707bd61","Type":"ContainerDied","Data":"7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40"} Oct 31 13:13:32.402663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.402626 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" event={"ID":"0c2e4a60-039e-4fef-af10-0ece9707bd61","Type":"ContainerDied","Data":"9c458de2c032af9d7b71f9f7fee8bdcf60e0080466fb662ac5374766ec09d9dd"} Oct 31 13:13:32.402820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.402804 5074 scope.go:122] "RemoveContainer" containerID="7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40" Oct 31 13:13:32.403100 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.403081 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2" Oct 31 13:13:32.446209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.446166 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2"] Oct 31 13:13:32.451665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.451619 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/auth-test-model-5452fc-kserve-7db9ff9785-klhs2"] Oct 31 13:13:32.454676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.454651 5074 scope.go:122] "RemoveContainer" containerID="7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40" Oct 31 13:13:32.455019 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:32.454996 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40\": container with ID starting with 7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40 not found: ID does not exist" containerID="7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40" Oct 31 13:13:32.455068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.455028 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40"} err="failed to get container status \"7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40\": rpc error: code = NotFound desc = could not find container \"7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40\": container with ID starting with 7c910581451e20a27adc463aba555174e7e2da8cc37abca9515d5482b889ef40 not found: ID does not exist" Oct 31 13:13:32.952866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.952835 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:13:32.953823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.953769 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:13:32.954485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:32.954460 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:13:33.201445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:33.201411 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:13:33.940138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:33.940106 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2e4a60-039e-4fef-af10-0ece9707bd61" path="/var/lib/kubelet/pods/0c2e4a60-039e-4fef-af10-0ece9707bd61/volumes" Oct 31 13:13:34.760953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:34.760915 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp"] Oct 31 13:13:34.761402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:34.761109 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" containerID="cri-o://b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b" gracePeriod=30 Oct 31 13:13:36.134675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.134642 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 13:13:36.229433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.229401 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-data\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.229611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.229494 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-token\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-token\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.229611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.229576 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-socket\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.229787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.229683 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-certs\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.229855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.229772 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-kube-api-access-vlfzv\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-kube-api-access-vlfzv\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.229921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.229872 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-credential-socket\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.230003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.229978 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-envoy\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.230140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.230117 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-data" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:36.230140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.230134 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-podinfo\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.230268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.230185 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istiod-ca-cert\") pod \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\" (UID: \"7d2f0b9c-1afb-4719-aee9-e73af34dbb26\") " Oct 31 13:13:36.230969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.230936 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.231405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.231372 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-socket" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:36.231616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.231592 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-certs" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:36.231616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.231603 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istiod-ca-cert" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:36.232993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.232962 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-credential-socket" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:36.234112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.234085 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-token" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:36.234573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.234543 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-podinfo" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:36.235782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.235759 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-kube-api-access-vlfzv" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "kube-api-access-vlfzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:36.236057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.236038 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-envoy" pod "7d2f0b9c-1afb-4719-aee9-e73af34dbb26" (UID: "7d2f0b9c-1afb-4719-aee9-e73af34dbb26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:36.332936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.332894 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.332936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.332936 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.333145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.332951 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.333145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.332965 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.333145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.332978 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.333145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.332992 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlfzv\" (UniqueName: \"kubernetes.io/projected/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-kube-api-access-vlfzv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.333145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.333010 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.333145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.333024 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2f0b9c-1afb-4719-aee9-e73af34dbb26-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:36.490288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.490247 5074 generic.go:356] "Generic (PLEG): container finished" podID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerID="4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41" exitCode=0 Oct 31 13:13:36.490504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.490305 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" Oct 31 13:13:36.490504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.490345 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" event={"ID":"7d2f0b9c-1afb-4719-aee9-e73af34dbb26","Type":"ContainerDied","Data":"4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41"} Oct 31 13:13:36.490504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.490417 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr" event={"ID":"7d2f0b9c-1afb-4719-aee9-e73af34dbb26","Type":"ContainerDied","Data":"1496f533ae26cb38eb55bb3f238232371ef33c9b2bdd5de13500d017564fc3b9"} Oct 31 13:13:36.490504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.490433 5074 scope.go:122] "RemoveContainer" containerID="4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41" Oct 31 13:13:36.504756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.504730 5074 scope.go:122] "RemoveContainer" containerID="4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41" Oct 31 13:13:36.505075 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:36.505053 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41\": container with ID starting with 4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41 not found: ID does not exist" containerID="4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41" Oct 31 13:13:36.505159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.505083 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41"} err="failed to get container status \"4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41\": rpc error: code = NotFound desc = could not find container \"4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41\": container with ID starting with 4962503e543926d3a0941d340e90f1b01299ce278f9cbb064c6460ea981c6e41 not found: ID does not exist" Oct 31 13:13:36.509732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.509704 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq"] Oct 31 13:13:36.509972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.509948 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" containerID="cri-o://7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0" gracePeriod=30 Oct 31 13:13:36.530941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.530909 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr"] Oct 31 13:13:36.536789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:36.536716 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-wb2hr"] Oct 31 13:13:37.356147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.356113 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 13:13:37.452177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.452140 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-certs\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.456795 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-kube-api-access-gdfrg\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-kube-api-access-gdfrg\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.456897 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istiod-ca-cert\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.456932 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-socket\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.456960 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-credential-socket\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.457121 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-data\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.457157 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-podinfo\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.457200 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-envoy\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.457244 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-token\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-token\") pod \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\" (UID: \"e38504a5-0bbf-485f-9a06-5e9964ff7b88\") " Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.455742 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-certs" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:37.459654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.457944 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-socket" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:37.460972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.460738 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istiod-ca-cert" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:37.461310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.461149 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-data" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:37.461739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.461433 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-credential-socket" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:37.465441 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.465398 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-podinfo" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:37.488661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.474065 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.489107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.489080 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.489209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.489198 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.524954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.489278 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.525103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.525084 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.525207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.525194 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.525301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.522693 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-kube-api-access-gdfrg" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "kube-api-access-gdfrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:37.529701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.522743 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-envoy" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:37.534706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.522838 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-token" pod "e38504a5-0bbf-485f-9a06-5e9964ff7b88" (UID: "e38504a5-0bbf-485f-9a06-5e9964ff7b88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:37.547670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.541918 5074 generic.go:356] "Generic (PLEG): container finished" podID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerID="ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae" exitCode=0 Oct 31 13:13:37.547670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.542029 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" Oct 31 13:13:37.547670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.542050 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" event={"ID":"e38504a5-0bbf-485f-9a06-5e9964ff7b88","Type":"ContainerDied","Data":"ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae"} Oct 31 13:13:37.547670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.546321 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn" event={"ID":"e38504a5-0bbf-485f-9a06-5e9964ff7b88","Type":"ContainerDied","Data":"3c6a7db5f5753c589082777aa752ef01cde341fb098c0e873edd21679f0a0756"} Oct 31 13:13:37.547670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.546361 5074 scope.go:122] "RemoveContainer" containerID="ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae" Oct 31 13:13:37.643672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.639797 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdfrg\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-kube-api-access-gdfrg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.643672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.639840 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.643672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.639867 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38504a5-0bbf-485f-9a06-5e9964ff7b88-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:37.659177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.659143 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn"] Oct 31 13:13:37.669159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.666848 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-847pn"] Oct 31 13:13:37.669159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.667061 5074 scope.go:122] "RemoveContainer" containerID="ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae" Oct 31 13:13:37.669159 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:37.667590 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae\": container with ID starting with ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae not found: ID does not exist" containerID="ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae" Oct 31 13:13:37.669159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.667628 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae"} err="failed to get container status \"ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae\": rpc error: code = NotFound desc = could not find container \"ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae\": container with ID starting with ada08db3da5e7651ffb251076f76bbf5ae00cca6f124f4a3f76236322391adae not found: ID does not exist" Oct 31 13:13:37.947070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.946992 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" path="/var/lib/kubelet/pods/7d2f0b9c-1afb-4719-aee9-e73af34dbb26/volumes" Oct 31 13:13:37.947533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:37.947517 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" path="/var/lib/kubelet/pods/e38504a5-0bbf-485f-9a06-5e9964ff7b88/volumes" Oct 31 13:13:40.066701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.066675 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 13:13:40.182242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182210 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-token\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-token\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182266 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-kube-api-access-dvkvx\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-kube-api-access-dvkvx\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182343 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-socket\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182403 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-data\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182451 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-podinfo\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182493 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-envoy\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182608 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-socket" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.182804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182620 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istiod-ca-cert\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182722 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-data" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.182858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182835 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-credential-socket\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182876 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-certs\") pod \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\" (UID: \"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb\") " Oct 31 13:13:40.182971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.182927 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istiod-ca-cert" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:40.183076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.183036 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-credential-socket" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.183252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.183224 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-certs" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.183720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.183689 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.183720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.183717 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.183856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.183732 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.183856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.183746 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.183856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.183760 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.184907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.184874 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-kube-api-access-dvkvx" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "kube-api-access-dvkvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:40.185039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.184986 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-token" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:40.185229 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.185208 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-envoy" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:40.185461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.185434 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-podinfo" pod "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" (UID: "b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:40.295143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.290850 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.295143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.290895 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkvx\" (UniqueName: \"kubernetes.io/projected/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-kube-api-access-dvkvx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.295143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.290913 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.295143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.290927 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:40.595174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.595084 5074 generic.go:356] "Generic (PLEG): container finished" podID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerID="b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b" exitCode=0 Oct 31 13:13:40.595174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.595148 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" Oct 31 13:13:40.595447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.595173 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" event={"ID":"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb","Type":"ContainerDied","Data":"b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b"} Oct 31 13:13:40.595447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.595247 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp" event={"ID":"b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb","Type":"ContainerDied","Data":"f8fef495678e8c1bd3c21a2837902e4e151f8d93f96cdc00b689f0a91c334a88"} Oct 31 13:13:40.595447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.595280 5074 scope.go:122] "RemoveContainer" containerID="b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b" Oct 31 13:13:40.608057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.608035 5074 scope.go:122] "RemoveContainer" containerID="b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b" Oct 31 13:13:40.608357 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:40.608335 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b\": container with ID starting with b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b not found: ID does not exist" containerID="b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b" Oct 31 13:13:40.608429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.608365 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b"} err="failed to get container status \"b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b\": rpc error: code = NotFound desc = could not find container \"b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b\": container with ID starting with b5e9c668093b666c6482e260601a52b7bc7ce1f1a3084153289a8b58b09e341b not found: ID does not exist" Oct 31 13:13:40.626820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.626621 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp"] Oct 31 13:13:40.630916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:40.630889 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-8hrlp"] Oct 31 13:13:41.803258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.803190 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 13:13:41.920687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.920625 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-certs\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.920903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.920718 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-podinfo\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.920903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.920771 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istiod-ca-cert\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.920903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.920804 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-kube-api-access-p54ps\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-kube-api-access-p54ps\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.920903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.920878 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-certs" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.921131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.920888 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-socket\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.921131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.920949 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-token\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-token\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.921255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.921182 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-credential-socket\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.921255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.921230 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-data\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.921366 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.921261 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-envoy\") pod \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\" (UID: \"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3\") " Oct 31 13:13:41.921459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.921413 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istiod-ca-cert" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:13:41.921618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.921596 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-socket" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.921758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.921738 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-credential-socket" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.921967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.921947 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-data" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.922323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.922297 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.922419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.922329 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.922419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.922343 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.922419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.922357 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.922419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.922373 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:41.923904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.923866 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-podinfo" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:13:41.924384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.924358 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-envoy" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:13:41.924825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.924799 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-kube-api-access-p54ps" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "kube-api-access-p54ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:41.926680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.925825 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-token" pod "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" (UID: "0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:13:41.987740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:41.985321 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" path="/var/lib/kubelet/pods/b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb/volumes" Oct 31 13:13:42.028402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.028372 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:42.033679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.033642 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:42.033679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.033680 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p54ps\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-kube-api-access-p54ps\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:42.033914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.033697 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:13:42.569463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.569427 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99"] Oct 31 13:13:42.570146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570127 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" Oct 31 13:13:42.570199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570147 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" Oct 31 13:13:42.570199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570182 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" Oct 31 13:13:42.570199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570188 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" Oct 31 13:13:42.570199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570196 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0c2e4a60-039e-4fef-af10-0ece9707bd61" containerName="main" Oct 31 13:13:42.570345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570201 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2e4a60-039e-4fef-af10-0ece9707bd61" containerName="main" Oct 31 13:13:42.570345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570208 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" Oct 31 13:13:42.570345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570215 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" Oct 31 13:13:42.570345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570223 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" Oct 31 13:13:42.570345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570227 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" Oct 31 13:13:42.570345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570340 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3222cd2-22f3-4c4b-8f9b-5e9ef73280eb" containerName="istio-proxy" Oct 31 13:13:42.570521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570352 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerName="istio-proxy" Oct 31 13:13:42.570521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570359 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2e4a60-039e-4fef-af10-0ece9707bd61" containerName="main" Oct 31 13:13:42.570521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570364 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38504a5-0bbf-485f-9a06-5e9964ff7b88" containerName="istio-proxy" Oct 31 13:13:42.570521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.570373 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2f0b9c-1afb-4719-aee9-e73af34dbb26" containerName="istio-proxy" Oct 31 13:13:42.574374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.574354 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.612653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.605224 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99"] Oct 31 13:13:42.625795 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.625755 5074 generic.go:356] "Generic (PLEG): container finished" podID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" containerID="7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0" exitCode=0 Oct 31 13:13:42.625980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.625832 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" Oct 31 13:13:42.625980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.625838 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" event={"ID":"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3","Type":"ContainerDied","Data":"7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0"} Oct 31 13:13:42.626106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.625997 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq" event={"ID":"0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3","Type":"ContainerDied","Data":"f592ee208b79d62c8ca04124578963feda7720614ed19b256dc74c89d54258ea"} Oct 31 13:13:42.626106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.626041 5074 scope.go:122] "RemoveContainer" containerID="7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0" Oct 31 13:13:42.642173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.641817 5074 scope.go:122] "RemoveContainer" containerID="7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0" Oct 31 13:13:42.642282 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:42.642190 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0\": container with ID starting with 7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0 not found: ID does not exist" containerID="7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0" Oct 31 13:13:42.642282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.642226 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0"} err="failed to get container status \"7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0\": rpc error: code = NotFound desc = could not find container \"7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0\": container with ID starting with 7164fd807f7e0b4b09b72f87da05994c96391baa16bff26136f5ebd3cb4752d0 not found: ID does not exist" Oct 31 13:13:42.653299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.653270 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq"] Oct 31 13:13:42.657391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.657367 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-qd7vq"] Oct 31 13:13:42.668158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.668137 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.668361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.668341 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.668408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.668379 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.668511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.668482 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.668561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.668526 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.668594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.668582 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhxj\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-kube-api-access-xfhxj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.668628 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.668608 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.769642 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.769605 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.769854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.769839 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.769921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.769882 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.769979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.769961 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.770045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.770023 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.770106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.770053 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhxj\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-kube-api-access-xfhxj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.770106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.770089 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.770740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.770708 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.772650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.772608 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.772964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.772946 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.773024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.772967 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.773290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.773273 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.783475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.783451 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.784021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.784000 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhxj\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-kube-api-access-xfhxj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8pb99\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:42.900521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:42.892911 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:43.697460 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:13:43.697416 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e0d99c_c28b_4b5c_be0e_d9e47fdf522f.slice/crio-7e9b203369a6d5500f29a74f147c00be2c4eb4912587ab1ee45a005369eb2ff7 WatchSource:0}: Error finding container 7e9b203369a6d5500f29a74f147c00be2c4eb4912587ab1ee45a005369eb2ff7: Status 404 returned error can't find the container with id 7e9b203369a6d5500f29a74f147c00be2c4eb4912587ab1ee45a005369eb2ff7 Oct 31 13:13:43.700004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:43.699428 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99"] Oct 31 13:13:43.713164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:43.702275 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:43.713164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:43.702359 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:43.939873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:43.939825 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3" path="/var/lib/kubelet/pods/0b6a25f2-5cdf-41e8-9600-2ee1a2f3c8a3/volumes" Oct 31 13:13:44.704657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:44.699417 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" event={"ID":"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f","Type":"ContainerStarted","Data":"733840636f3817b17b96f3f0ef0f72d72f75a262f1bf9fb05268e5601f508997"} Oct 31 13:13:44.704657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:44.699491 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" event={"ID":"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f","Type":"ContainerStarted","Data":"7e9b203369a6d5500f29a74f147c00be2c4eb4912587ab1ee45a005369eb2ff7"} Oct 31 13:13:44.716660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:44.713717 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:44.724166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:44.723818 5074 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8pb99 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:13:44.724166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:44.723879 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" podUID="02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:13:44.742861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:44.742802 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" podStartSLOduration=2.742783396 podStartE2EDuration="2.742783396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:44.738039713 +0000 UTC m=+2518.915757699" watchObservedRunningTime="2026-10-31 13:13:44.742783396 +0000 UTC m=+2518.920501383" Oct 31 13:13:45.710351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:45.710321 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:13:47.870838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:47.870797 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f"] Oct 31 13:13:47.877581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:47.877557 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:47.883399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:47.883360 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-4574ec-openshift-default-dockercfg-kg8h4\"" Oct 31 13:13:47.893698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:47.890127 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f"] Oct 31 13:13:48.020118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020081 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smkq\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-kube-api-access-8smkq\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020147 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020187 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020238 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020323 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020350 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020381 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020397 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.020544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.020506 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122009 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smkq\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-kube-api-access-8smkq\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122092 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122109 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122181 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122226 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122274 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122360 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122394 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122505 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.122777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.122624 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-certs\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.123036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.123004 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istiod-ca-cert\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.123138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.123048 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-credential-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.123138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.123092 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-socket\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.123402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.123378 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-data\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.124840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.124816 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-envoy\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.124960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.124944 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-podinfo\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.136024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.135993 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smkq\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-kube-api-access-8smkq\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.136120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.136026 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-token\") pod \"e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.213663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.213097 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:48.550352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.550318 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f"] Oct 31 13:13:48.550819 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:13:48.550795 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e86b90_dc70_4ef8_9ff5_fcfd8c9970bd.slice/crio-7442a1387646a6c31b7e0a773c9824b6fa846bb25487cdfccc0081c7e424d04e WatchSource:0}: Error finding container 7442a1387646a6c31b7e0a773c9824b6fa846bb25487cdfccc0081c7e424d04e: Status 404 returned error can't find the container with id 7442a1387646a6c31b7e0a773c9824b6fa846bb25487cdfccc0081c7e424d04e Oct 31 13:13:48.554337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.554296 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:48.554437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.554388 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:48.554504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.554436 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:13:48.739327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.739282 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" event={"ID":"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd","Type":"ContainerStarted","Data":"e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d"} Oct 31 13:13:48.739500 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.739418 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" event={"ID":"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd","Type":"ContainerStarted","Data":"7442a1387646a6c31b7e0a773c9824b6fa846bb25487cdfccc0081c7e424d04e"} Oct 31 13:13:48.770262 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:48.770198 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podStartSLOduration=1.7701798709999998 podStartE2EDuration="1.770179871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:48.764197299 +0000 UTC m=+2522.941915296" watchObservedRunningTime="2026-10-31 13:13:48.770179871 +0000 UTC m=+2522.947897856" Oct 31 13:13:49.215276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:49.215163 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:49.217512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:49.217476 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:13:49.217680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:49.217543 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 31 13:13:50.138352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.138318 5074 scope.go:122] "RemoveContainer" containerID="c283118cc3a53190cfd205d82b3c04755720c5a0ba147c09eda5976df93110f7" Oct 31 13:13:50.149558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.149535 5074 scope.go:122] "RemoveContainer" containerID="761d3d2b85d4217d2376ab1426320d611401f7cce1d72ad6af83176e8352e415" Oct 31 13:13:50.159160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.159141 5074 scope.go:122] "RemoveContainer" containerID="ddab2006f9f44d11cb90f7bc4e76ad65e0b0bae923aff3fd9e548f8c758eb9dd" Oct 31 13:13:50.168516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.168498 5074 scope.go:122] "RemoveContainer" containerID="bac276f6cc9a320e40ebd75835509c24c13bb47ef38657c4e28c3ab5158fe792" Oct 31 13:13:50.212851 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.212727 5074 scope.go:122] "RemoveContainer" containerID="3057e0235f467d873b1fd18f95f64004acc1107681ec8581835007eb376edf43" Oct 31 13:13:50.213999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.213964 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:13:50.214112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.214030 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 31 13:13:50.223588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.223565 5074 scope.go:122] "RemoveContainer" containerID="a93cd24576e2db37f764b8c67e043a2c8d0ba7470d41698709b25143ef5af1a3" Oct 31 13:13:50.232955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.232936 5074 scope.go:122] "RemoveContainer" containerID="82b751d33530e2be1cbf55ffed130e665257e2331de25c08a6eda31da096886b" Oct 31 13:13:50.242166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.242143 5074 scope.go:122] "RemoveContainer" containerID="077965c12547aafb404f30ae5ec83fabb9d14bdb875878d0371a638bfc4ea335" Oct 31 13:13:50.254034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.254015 5074 scope.go:122] "RemoveContainer" containerID="f73d8f08dac958c0fab7c3b21823d93144cd1c41b08b48ff7a128ae0c3686e7e" Oct 31 13:13:50.263473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:50.263439 5074 scope.go:122] "RemoveContainer" containerID="9e48c0c7368bb823a69abb98bb20069f0b6e597dc948943cdb4791382da73210" Oct 31 13:13:51.214964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:51.214925 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:13:51.215185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:51.214989 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 31 13:13:52.227187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:52.227089 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:13:52.227726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:52.227159 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 31 13:13:53.229061 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:53.229021 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:13:53.230793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:53.230756 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 31 13:13:53.694715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:53.690816 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:13:54.214948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:54.214909 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:13:54.215164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:54.214973 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 31 13:13:55.220658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:55.214182 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 31 13:13:55.220658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:55.214259 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 31 13:13:55.987018 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:13:55.986974 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:13:56.219664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:56.219603 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:56.220184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:56.220155 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:13:56.221002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:13:56.220980 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:14:10.097654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.097452 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x"] Oct 31 13:14:10.108701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.106744 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.119491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.119379 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c2c98b-dockercfg-pdrmw\"" Oct 31 13:14:10.137893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.137860 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x"] Oct 31 13:14:10.163775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.163738 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c2c98b\"" Oct 31 13:14:10.223652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.222948 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4j5\" (UniqueName: \"kubernetes.io/projected/6eb5dffd-8093-45ec-aada-8e485a60b178-kube-api-access-nt4j5\") pod \"payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.223652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.223022 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb5dffd-8093-45ec-aada-8e485a60b178-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.338658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.337666 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4j5\" (UniqueName: \"kubernetes.io/projected/6eb5dffd-8093-45ec-aada-8e485a60b178-kube-api-access-nt4j5\") pod \"payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.338658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.337750 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb5dffd-8093-45ec-aada-8e485a60b178-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.342655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.338922 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb5dffd-8093-45ec-aada-8e485a60b178-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.406715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.377924 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4j5\" (UniqueName: \"kubernetes.io/projected/6eb5dffd-8093-45ec-aada-8e485a60b178-kube-api-access-nt4j5\") pod \"payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.440652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.435241 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:10.769216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:10.769174 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x"] Oct 31 13:14:10.774433 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:14:10.774400 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb5dffd_8093_45ec_aada_8e485a60b178.slice/crio-25ca39c166ec34460924fa3e0d262ebd42ef6e0ea931ebd12c8507a7464cb5b4 WatchSource:0}: Error finding container 25ca39c166ec34460924fa3e0d262ebd42ef6e0ea931ebd12c8507a7464cb5b4: Status 404 returned error can't find the container with id 25ca39c166ec34460924fa3e0d262ebd42ef6e0ea931ebd12c8507a7464cb5b4 Oct 31 13:14:11.161077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.160979 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" event={"ID":"6eb5dffd-8093-45ec-aada-8e485a60b178","Type":"ContainerStarted","Data":"1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9"} Oct 31 13:14:11.161077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.161053 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:11.161538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.161080 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" event={"ID":"6eb5dffd-8093-45ec-aada-8e485a60b178","Type":"ContainerStarted","Data":"25ca39c166ec34460924fa3e0d262ebd42ef6e0ea931ebd12c8507a7464cb5b4"} Oct 31 13:14:11.288661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.288508 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" podStartSLOduration=1.288488958 podStartE2EDuration="1.288488958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:11.288037367 +0000 UTC m=+2545.465755352" watchObservedRunningTime="2026-10-31 13:14:11.288488958 +0000 UTC m=+2545.466206944" Oct 31 13:14:11.769266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.769231 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx"] Oct 31 13:14:11.775616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.775576 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.806322 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.806283 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx"] Oct 31 13:14:11.821664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.821145 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5dc570-openshift-default-dockercfg-dvsb2\"" Oct 31 13:14:11.889212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889161 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889235 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889302 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kzvn\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-kube-api-access-4kzvn\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889336 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889391 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889420 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18c2078-ee42-451c-9685-1e6f118ac9ec-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889622 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889765 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.889872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.889818 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.990942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.990908 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991044 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991095 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991173 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991213 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991282 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kzvn\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-kube-api-access-4kzvn\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991315 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991373 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991409 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18c2078-ee42-451c-9685-1e6f118ac9ec-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991508 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.991663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.991621 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.992066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.992037 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.992199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.992152 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18c2078-ee42-451c-9685-1e6f118ac9ec-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.992296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.992275 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.993432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.993409 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:11.994124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:11.994100 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:12.006784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:12.006755 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:12.006918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:12.006854 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kzvn\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-kube-api-access-4kzvn\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:12.110138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:12.108834 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:12.762169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:12.761767 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx"] Oct 31 13:14:12.771887 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:14:12.771791 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18c2078_ee42_451c_9685_1e6f118ac9ec.slice/crio-91e7131edae04fd17c279e7a07e39e002078dc48c0110f2e9b159633a769872c WatchSource:0}: Error finding container 91e7131edae04fd17c279e7a07e39e002078dc48c0110f2e9b159633a769872c: Status 404 returned error can't find the container with id 91e7131edae04fd17c279e7a07e39e002078dc48c0110f2e9b159633a769872c Oct 31 13:14:12.773691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:12.773627 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:12.773820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:12.773744 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:12.773881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:12.773825 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:13.232953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:13.231296 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" event={"ID":"c18c2078-ee42-451c-9685-1e6f118ac9ec","Type":"ContainerStarted","Data":"eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591"} Oct 31 13:14:13.232953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:13.231362 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" event={"ID":"c18c2078-ee42-451c-9685-1e6f118ac9ec","Type":"ContainerStarted","Data":"91e7131edae04fd17c279e7a07e39e002078dc48c0110f2e9b159633a769872c"} Oct 31 13:14:13.271741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:13.271672 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podStartSLOduration=2.271651497 podStartE2EDuration="2.271651497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:13.257807534 +0000 UTC m=+2547.435525519" watchObservedRunningTime="2026-10-31 13:14:13.271651497 +0000 UTC m=+2547.449369481" Oct 31 13:14:14.110046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:14.110003 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:14.111803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:14.111772 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 13:14:14.111950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:14.111826 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 13:14:15.110542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:15.110489 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 13:14:15.111051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:15.110569 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 13:14:16.109948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:16.109909 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 13:14:16.110122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:16.109970 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 13:14:17.115792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:17.115755 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 13:14:17.120578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:17.116346 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 13:14:18.110396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:18.110356 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 13:14:18.110731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:18.110696 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 13:14:19.109623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:19.109576 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 13:14:19.110170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:19.109663 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 13:14:20.117962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:20.117927 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:20.118461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:20.118413 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:20.119266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:20.119237 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:14:22.220939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:22.220905 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:14:26.063391 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:14:26.063301 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:14:29.279675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:29.279618 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:14:55.383095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.383055 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t"] Oct 31 13:14:55.389373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.389324 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.397244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.397214 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t"] Oct 31 13:14:55.408719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.408692 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-e4da4b834193\"" Oct 31 13:14:55.409425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.409398 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb\"" Oct 31 13:14:55.409567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.409488 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"routing-overlay\"" Oct 31 13:14:55.409671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.409567 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"openshift-service-ca.crt\"" Oct 31 13:14:55.409671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.409647 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"kube-root-ca.crt\"" Oct 31 13:14:55.409792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.409752 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-e2e--9527f114-dockercfg-5ljgf\"" Oct 31 13:14:55.455802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.455763 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42tgh\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-kube-api-access-42tgh\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.456041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.455836 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.456041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.455896 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.456041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.455926 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.634368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.632037 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42tgh\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-kube-api-access-42tgh\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.634368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.632112 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.634368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.632168 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.634368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.632194 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.634368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.633106 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.634368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.633795 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.687201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.671897 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.701657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.698064 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42tgh\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-kube-api-access-42tgh\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.727550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:55.727111 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:55.997046 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:14:55.997002 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:14:56.091514 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:14:56.091483 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d10a9ea_ccde_4cc8_83fb_8f5ed46a7e51.slice/crio-fbec6c87e1224309c2ae80f8c0adba23dd50b03363aaf7c34ba198a0dc2abe0f WatchSource:0}: Error finding container fbec6c87e1224309c2ae80f8c0adba23dd50b03363aaf7c34ba198a0dc2abe0f: Status 404 returned error can't find the container with id fbec6c87e1224309c2ae80f8c0adba23dd50b03363aaf7c34ba198a0dc2abe0f Oct 31 13:14:56.091997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:56.091970 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t"] Oct 31 13:14:56.785379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:56.785340 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" event={"ID":"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51","Type":"ContainerStarted","Data":"3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404"} Oct 31 13:14:56.785848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:56.785409 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" event={"ID":"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51","Type":"ContainerStarted","Data":"fbec6c87e1224309c2ae80f8c0adba23dd50b03363aaf7c34ba198a0dc2abe0f"} Oct 31 13:14:56.785848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:56.785437 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:14:56.846916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:56.846860 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" podStartSLOduration=1.846845606 podStartE2EDuration="1.846845606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:56.844705405 +0000 UTC m=+2591.022423389" watchObservedRunningTime="2026-10-31 13:14:56.846845606 +0000 UTC m=+2591.024563590" Oct 31 13:14:56.957761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:56.957725 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx"] Oct 31 13:14:56.962267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:56.962241 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.033709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.020853 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx"] Oct 31 13:14:57.033709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.031970 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5"] Oct 31 13:14:57.039858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.039786 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.072501 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072458 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rxs\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-kube-api-access-b4rxs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.072767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072524 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.072767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072562 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7pg\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-kube-api-access-lb7pg\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.072911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072787 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6546233-b00c-44c9-9d2a-e3078540b475-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.072911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072868 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.072911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072900 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.073078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072937 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6546233-b00c-44c9-9d2a-e3078540b475-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.073078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072960 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.073078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.072994 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.073078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.073047 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.073292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.073088 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.073292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.073140 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.073292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.073178 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.073292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.073202 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.073292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.073236 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.091864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.091248 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.091864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.091363 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.091864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.091429 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.110760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.110677 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5"] Oct 31 13:14:57.193669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193607 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6546233-b00c-44c9-9d2a-e3078540b475-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.193877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193739 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.193877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193780 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.193877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193820 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6546233-b00c-44c9-9d2a-e3078540b475-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.193877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193847 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193883 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193940 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.193991 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194047 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194089 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194113 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194144 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.194240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194169 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194275 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194350 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194416 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rxs\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-kube-api-access-b4rxs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.194784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194458 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194491 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7pg\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-kube-api-access-lb7pg\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.194784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194507 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6546233-b00c-44c9-9d2a-e3078540b475-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.195158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194927 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.195158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.194975 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.195272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.195186 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-data\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.195272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.195234 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.195724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.195578 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.195848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.195817 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istiod-ca-cert\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.196432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.196193 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-credential-socket\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.196432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.196355 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.197245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.197227 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-certs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.198000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.197967 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.199452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.199410 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6546233-b00c-44c9-9d2a-e3078540b475-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.199690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.199627 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-envoy\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.200921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.200882 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-podinfo\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.214339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.214173 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.214339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.214196 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-istio-token\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.214339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.214298 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rxs\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-kube-api-access-b4rxs\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.226899 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.220310 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7pg\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-kube-api-access-lb7pg\") pod \"e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.304693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.304584 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:57.404378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.404337 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:57.609427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.609336 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx"] Oct 31 13:14:57.615122 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:14:57.615082 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6546233_b00c_44c9_9d2a_e3078540b475.slice/crio-57cbe2c726191d820a53802739ed13d5e3d35ea00f299668b6daa437b364a7b5 WatchSource:0}: Error finding container 57cbe2c726191d820a53802739ed13d5e3d35ea00f299668b6daa437b364a7b5: Status 404 returned error can't find the container with id 57cbe2c726191d820a53802739ed13d5e3d35ea00f299668b6daa437b364a7b5 Oct 31 13:14:57.618473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.618428 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:57.618553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.618526 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:57.618612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.618595 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:57.733840 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:14:57.733806 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae025a9_1bf1_46b7_96c3_b0e0cd679388.slice/crio-4e89d4d686dd39a6bc8989188037f735451888d83bbc6e62e0e9c421e998a985 WatchSource:0}: Error finding container 4e89d4d686dd39a6bc8989188037f735451888d83bbc6e62e0e9c421e998a985: Status 404 returned error can't find the container with id 4e89d4d686dd39a6bc8989188037f735451888d83bbc6e62e0e9c421e998a985 Oct 31 13:14:57.736183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.736156 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5"] Oct 31 13:14:57.737020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.736986 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:57.737153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.737051 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:57.737153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.737078 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:14:57.809300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.809227 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" event={"ID":"8ae025a9-1bf1-46b7-96c3-b0e0cd679388","Type":"ContainerStarted","Data":"4e89d4d686dd39a6bc8989188037f735451888d83bbc6e62e0e9c421e998a985"} Oct 31 13:14:57.813469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.813440 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" event={"ID":"f6546233-b00c-44c9-9d2a-e3078540b475","Type":"ContainerStarted","Data":"b4e65cfd2d3ed753425c46b86a735de2aea7c6844886ee1bffd5893e17b8eb05"} Oct 31 13:14:57.813601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.813491 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" event={"ID":"f6546233-b00c-44c9-9d2a-e3078540b475","Type":"ContainerStarted","Data":"57cbe2c726191d820a53802739ed13d5e3d35ea00f299668b6daa437b364a7b5"} Oct 31 13:14:57.850439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:57.850383 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podStartSLOduration=1.850367632 podStartE2EDuration="1.850367632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:57.842238742 +0000 UTC m=+2592.019956728" watchObservedRunningTime="2026-10-31 13:14:57.850367632 +0000 UTC m=+2592.028085618" Oct 31 13:14:58.305714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:58.305674 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:14:58.307084 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:58.307057 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:14:58.307293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:58.307264 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:14:58.866393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:58.866352 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" event={"ID":"8ae025a9-1bf1-46b7-96c3-b0e0cd679388","Type":"ContainerStarted","Data":"a99bd2295601afa7c89b6d17e4df36fe4462bdc649fc5ec4ce2eb0ba7379b012"} Oct 31 13:14:58.939229 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:58.936668 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podStartSLOduration=2.936646629 podStartE2EDuration="2.936646629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:58.923007714 +0000 UTC m=+2593.100725741" watchObservedRunningTime="2026-10-31 13:14:58.936646629 +0000 UTC m=+2593.114364608" Oct 31 13:14:59.318656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:59.318082 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:14:59.318656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:59.318150 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:14:59.418528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:59.415295 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:14:59.428010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:59.421397 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 13:14:59.428010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:14:59.421453 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 13:15:00.248166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.230857 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8"] Oct 31 13:15:00.259782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.255756 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46"] Oct 31 13:15:00.259782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.256737 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:00.355658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.347012 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:15:00.466690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.399534 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:15:00.544312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.449804 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:15:00.629678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.628845 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abcd738-c940-44ba-9b90-6081d4a6b861-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:00.629678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.629018 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqc5\" (UniqueName: \"kubernetes.io/projected/1abcd738-c940-44ba-9b90-6081d4a6b861-kube-api-access-mxqc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:00.662699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.654068 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 13:15:00.668573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.654138 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 13:15:00.668573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.668264 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:15:00.668573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.668315 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:15:00.668883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.668752 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s"] Oct 31 13:15:00.730879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.730841 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abcd738-c940-44ba-9b90-6081d4a6b861-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:00.731278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.731254 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqc5\" (UniqueName: \"kubernetes.io/projected/1abcd738-c940-44ba-9b90-6081d4a6b861-kube-api-access-mxqc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:00.732255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.732228 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:00.739407 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.739023 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abcd738-c940-44ba-9b90-6081d4a6b861-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:00.755798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.755758 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m"] Oct 31 13:15:00.757273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.757247 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqc5\" (UniqueName: \"kubernetes.io/projected/1abcd738-c940-44ba-9b90-6081d4a6b861-kube-api-access-mxqc5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:00.757555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.757532 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:00.763197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.763169 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm"] Oct 31 13:15:00.763496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.763472 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:00.776656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.774790 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s"] Oct 31 13:15:00.776656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.774922 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8"] Oct 31 13:15:00.776656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.774948 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46"] Oct 31 13:15:00.776656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.774973 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m"] Oct 31 13:15:00.776656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.774998 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm"] Oct 31 13:15:00.776656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.774873 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:00.826660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.826461 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d"] Oct 31 13:15:00.836706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.836665 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g78kc\" (UniqueName: \"kubernetes.io/projected/6a434213-99ab-4ceb-8a6e-424927391e99-kube-api-access-g78kc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:00.836876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.836789 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmpk\" (UniqueName: \"kubernetes.io/projected/dddae608-97bc-47b7-882d-710474720e95-kube-api-access-vsmpk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:00.836947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.836909 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:00.837039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.837019 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddae608-97bc-47b7-882d-710474720e95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:00.837217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.837195 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a434213-99ab-4ceb-8a6e-424927391e99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:00.837291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.837281 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwdc\" (UniqueName: \"kubernetes.io/projected/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-kube-api-access-8jwdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:00.839820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.839793 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96"] Oct 31 13:15:00.840787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.840765 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:00.846328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.846307 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4"] Oct 31 13:15:00.846433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.846416 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:00.850991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.850976 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t"] Oct 31 13:15:00.851267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.851248 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:00.855773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.855753 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk"] Oct 31 13:15:00.855928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.855917 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:00.861417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.861396 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj"] Oct 31 13:15:00.861543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.861521 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:00.867254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.867239 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d"] Oct 31 13:15:00.867387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.867345 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:00.877038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.877021 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4"] Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944599 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944678 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g78kc\" (UniqueName: \"kubernetes.io/projected/6a434213-99ab-4ceb-8a6e-424927391e99-kube-api-access-g78kc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944776 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmpk\" (UniqueName: \"kubernetes.io/projected/dddae608-97bc-47b7-882d-710474720e95-kube-api-access-vsmpk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944831 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721a5252-6907-4590-a6fc-bd7e1dc2a780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944860 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7n7\" (UniqueName: \"kubernetes.io/projected/721a5252-6907-4590-a6fc-bd7e1dc2a780-kube-api-access-cx7n7\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944884 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944908 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:00.944982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.944975 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pl2\" (UniqueName: \"kubernetes.io/projected/095b3e51-cd04-44ef-9798-fe0e94c19bef-kube-api-access-g6pl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945001 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbrt\" (UniqueName: \"kubernetes.io/projected/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-kube-api-access-6mbrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945023 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddae608-97bc-47b7-882d-710474720e95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945048 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9v8\" (UniqueName: \"kubernetes.io/projected/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-kube-api-access-cn9v8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945092 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a45b88-fafa-420f-a7ff-99abed7bd013-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945117 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9x8\" (UniqueName: \"kubernetes.io/projected/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-kube-api-access-px9x8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945146 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945163 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98qt\" (UniqueName: \"kubernetes.io/projected/03a45b88-fafa-420f-a7ff-99abed7bd013-kube-api-access-p98qt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945178 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945236 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a434213-99ab-4ceb-8a6e-424927391e99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945269 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqc7\" (UniqueName: \"kubernetes.io/projected/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-kube-api-access-5lqc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945293 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwdc\" (UniqueName: \"kubernetes.io/projected/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-kube-api-access-8jwdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:00.945574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.945335 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b3e51-cd04-44ef-9798-fe0e94c19bef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:00.947017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.946537 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:00.947017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.946714 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddae608-97bc-47b7-882d-710474720e95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:00.948420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.948376 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a434213-99ab-4ceb-8a6e-424927391e99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:00.950386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.950362 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv"] Oct 31 13:15:00.956824 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.956802 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t"] Oct 31 13:15:00.956931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.956902 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:00.959880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.959857 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g78kc\" (UniqueName: \"kubernetes.io/projected/6a434213-99ab-4ceb-8a6e-424927391e99-kube-api-access-g78kc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:00.960099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.960069 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96"] Oct 31 13:15:00.960843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.960824 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmpk\" (UniqueName: \"kubernetes.io/projected/dddae608-97bc-47b7-882d-710474720e95-kube-api-access-vsmpk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:00.964010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.963984 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk"] Oct 31 13:15:00.967438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.967393 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwdc\" (UniqueName: \"kubernetes.io/projected/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-kube-api-access-8jwdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:00.991454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:00.969158 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt"] Oct 31 13:15:01.011277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.011246 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:01.021036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.020786 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:01.131766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.131034 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm"] Oct 31 13:15:01.156673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.153887 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:01.165046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.164990 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a45b88-fafa-420f-a7ff-99abed7bd013-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:01.165123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165065 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9x8\" (UniqueName: \"kubernetes.io/projected/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-kube-api-access-px9x8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:01.165517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165341 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:01.165517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165490 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:01.165785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165538 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98qt\" (UniqueName: \"kubernetes.io/projected/03a45b88-fafa-420f-a7ff-99abed7bd013-kube-api-access-p98qt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:01.165785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165573 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:01.165785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165715 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7399962-c419-4d22-85f0-6364c37e8379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:01.165785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165751 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqc7\" (UniqueName: \"kubernetes.io/projected/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-kube-api-access-5lqc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:01.165992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165839 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b3e51-cd04-44ef-9798-fe0e94c19bef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:01.165992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.165954 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:01.166099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166053 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjslw\" (UniqueName: \"kubernetes.io/projected/d7399962-c419-4d22-85f0-6364c37e8379-kube-api-access-xjslw\") pod \"maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:01.166152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166093 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56sr\" (UniqueName: \"kubernetes.io/projected/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-kube-api-access-t56sr\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:01.166265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166208 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721a5252-6907-4590-a6fc-bd7e1dc2a780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:01.166330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166261 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7n7\" (UniqueName: \"kubernetes.io/projected/721a5252-6907-4590-a6fc-bd7e1dc2a780-kube-api-access-cx7n7\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:01.166330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166320 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:01.168285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166448 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pl2\" (UniqueName: \"kubernetes.io/projected/095b3e51-cd04-44ef-9798-fe0e94c19bef-kube-api-access-g6pl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:01.168285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166502 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbrt\" (UniqueName: \"kubernetes.io/projected/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-kube-api-access-6mbrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:01.168285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.166562 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9v8\" (UniqueName: \"kubernetes.io/projected/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-kube-api-access-cn9v8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:01.168285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.168094 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b3e51-cd04-44ef-9798-fe0e94c19bef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:01.208662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.169240 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:01.252957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.178992 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:01.253365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.179289 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:01.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.180149 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a45b88-fafa-420f-a7ff-99abed7bd013-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:01.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.182414 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721a5252-6907-4590-a6fc-bd7e1dc2a780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:01.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.184036 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:01.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.194300 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:01.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.194972 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:01.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.252754 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj"] Oct 31 13:15:01.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.254118 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4"] Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.309234 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjslw\" (UniqueName: \"kubernetes.io/projected/d7399962-c419-4d22-85f0-6364c37e8379-kube-api-access-xjslw\") pod \"maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.309299 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56sr\" (UniqueName: \"kubernetes.io/projected/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-kube-api-access-t56sr\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.309597 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.309730 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7399962-c419-4d22-85f0-6364c37e8379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.313106 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98qt\" (UniqueName: \"kubernetes.io/projected/03a45b88-fafa-420f-a7ff-99abed7bd013-kube-api-access-p98qt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.314373 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7n7\" (UniqueName: \"kubernetes.io/projected/721a5252-6907-4590-a6fc-bd7e1dc2a780-kube-api-access-cx7n7\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.252884 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.322465 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.325336 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9v8\" (UniqueName: \"kubernetes.io/projected/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-kube-api-access-cn9v8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:01.338189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.331032 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9x8\" (UniqueName: \"kubernetes.io/projected/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-kube-api-access-px9x8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:01.347769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.345664 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56sr\" (UniqueName: \"kubernetes.io/projected/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-kube-api-access-t56sr\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:01.347769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.346938 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqc7\" (UniqueName: \"kubernetes.io/projected/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-kube-api-access-5lqc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:01.359699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.348501 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:01.359699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.352522 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k"] Oct 31 13:15:01.361451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.361223 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7399962-c419-4d22-85f0-6364c37e8379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:01.361451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.361374 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjslw\" (UniqueName: \"kubernetes.io/projected/d7399962-c419-4d22-85f0-6364c37e8379-kube-api-access-xjslw\") pod \"maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:01.365970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.365913 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbrt\" (UniqueName: \"kubernetes.io/projected/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-kube-api-access-6mbrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:01.368138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.368112 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:01.458205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.458163 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:15:01.458402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.458237 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:15:01.462668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.462182 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28505a68-5e10-4461-a95e-5680564eb5b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:01.462668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.462565 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n66b5\" (UniqueName: \"kubernetes.io/projected/28505a68-5e10-4461-a95e-5680564eb5b7-kube-api-access-n66b5\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:01.462668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.462604 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxb7\" (UniqueName: \"kubernetes.io/projected/2af9234f-cf40-4883-8d91-c392be69d425-kube-api-access-bnxb7\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:01.463262 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.463000 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af9234f-cf40-4883-8d91-c392be69d425-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:01.465537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.465519 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:01.520397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.481799 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp"] Oct 31 13:15:01.520397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.489440 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 13:15:01.520397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.489492 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 13:15:01.551660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.550753 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:01.552979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.552948 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.559494 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.561408 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pl2\" (UniqueName: \"kubernetes.io/projected/095b3e51-cd04-44ef-9798-fe0e94c19bef-kube-api-access-g6pl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.564815 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n66b5\" (UniqueName: \"kubernetes.io/projected/28505a68-5e10-4461-a95e-5680564eb5b7-kube-api-access-n66b5\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.564872 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxb7\" (UniqueName: \"kubernetes.io/projected/2af9234f-cf40-4883-8d91-c392be69d425-kube-api-access-bnxb7\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.564942 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af9234f-cf40-4883-8d91-c392be69d425-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.565051 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzb6\" (UniqueName: \"kubernetes.io/projected/af8fd503-9169-4d5a-aab0-799659a0be24-kube-api-access-rlzb6\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.565149 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28505a68-5e10-4461-a95e-5680564eb5b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.565349 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8fd503-9169-4d5a-aab0-799659a0be24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.573567 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af9234f-cf40-4883-8d91-c392be69d425-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.580351 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28505a68-5e10-4461-a95e-5680564eb5b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.595277 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt"] Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.595360 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.595436 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4"] Oct 31 13:15:01.695335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.639822 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:01.725053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.710690 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp"] Oct 31 13:15:01.800655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.761044 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshdz\" (UniqueName: \"kubernetes.io/projected/f659ac7d-4b1e-429b-8f44-d41fab077a70-kube-api-access-pshdz\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:01.800655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.761115 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8fd503-9169-4d5a-aab0-799659a0be24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:01.800655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.761205 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f659ac7d-4b1e-429b-8f44-d41fab077a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:01.800655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.761505 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzb6\" (UniqueName: \"kubernetes.io/projected/af8fd503-9169-4d5a-aab0-799659a0be24-kube-api-access-rlzb6\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:01.816987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.816449 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzb6\" (UniqueName: \"kubernetes.io/projected/af8fd503-9169-4d5a-aab0-799659a0be24-kube-api-access-rlzb6\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:01.816987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.816761 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:01.827374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.824521 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8fd503-9169-4d5a-aab0-799659a0be24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:01.828119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.828085 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxb7\" (UniqueName: \"kubernetes.io/projected/2af9234f-cf40-4883-8d91-c392be69d425-kube-api-access-bnxb7\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:01.847994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.847959 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k"] Oct 31 13:15:01.860762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.860728 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm"] Oct 31 13:15:01.867759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.867728 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv"] Oct 31 13:15:01.877761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.866587 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n66b5\" (UniqueName: \"kubernetes.io/projected/28505a68-5e10-4461-a95e-5680564eb5b7-kube-api-access-n66b5\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:01.887666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.882847 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshdz\" (UniqueName: \"kubernetes.io/projected/f659ac7d-4b1e-429b-8f44-d41fab077a70-kube-api-access-pshdz\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:01.887666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.882939 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f659ac7d-4b1e-429b-8f44-d41fab077a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:01.887666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:01.884170 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f659ac7d-4b1e-429b-8f44-d41fab077a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:02.091389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.091348 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:02.184663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.174368 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshdz\" (UniqueName: \"kubernetes.io/projected/f659ac7d-4b1e-429b-8f44-d41fab077a70-kube-api-access-pshdz\") pod \"maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:02.232359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.232107 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:02.463939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.463896 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 13:15:02.464595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.464563 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:02.608867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.608828 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 13:15:02.616749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.616705 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 13:15:02.639174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.634188 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 31 13:15:02.639174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.636805 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:15:02.639174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.636852 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:15:02.764681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.753736 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:02.841860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.830123 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:02.843083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.843053 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:02.850166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.850140 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:02.913686 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:02.913458 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a434213_99ab_4ceb_8a6e_424927391e99.slice/crio-6372dad55f4a22e9ebae0f5227b15718ad6ee4f1c7831c2c42180d8af2867364 WatchSource:0}: Error finding container 6372dad55f4a22e9ebae0f5227b15718ad6ee4f1c7831c2c42180d8af2867364: Status 404 returned error can't find the container with id 6372dad55f4a22e9ebae0f5227b15718ad6ee4f1c7831c2c42180d8af2867364 Oct 31 13:15:02.926181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:02.926146 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 13:15:03.030589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.030549 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:03.050295 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.050262 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:03.172838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.172793 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:03.173933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.173900 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:15:03.252731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.244125 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 13:15:03.337665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.331000 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:03.337665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.332670 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:15:03.337665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.332729 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:15:03.391087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.387992 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 13:15:03.508757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.491789 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 13:15:03.508757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.491847 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 13:15:03.760500 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.759720 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:03.760500 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:03.760001 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:15:04.203663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.203306 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:15:04.458805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.455249 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:15:04.458805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.455325 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:15:04.460803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.460697 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 13:15:04.491657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.466967 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 13:15:04.589707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.551763 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 13:15:04.589707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.568820 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:15:04.589707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.568915 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:15:04.589707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.568968 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 13:15:04.589707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.569024 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:15:04.861760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:04.861718 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 13:15:05.118261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.072884 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" event={"ID":"6a434213-99ab-4ceb-8a6e-424927391e99","Type":"ContainerStarted","Data":"6372dad55f4a22e9ebae0f5227b15718ad6ee4f1c7831c2c42180d8af2867364"} Oct 31 13:15:05.267539 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.267496 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerStarted","Data":"fa69467fb4526ba62a439fc6d9107f70f9673bf3781177f23c912cb286597f85"} Oct 31 13:15:05.348774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.348684 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46"] Oct 31 13:15:05.351547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.351514 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 13:15:05.435468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.435377 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 13:15:05.436107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.395625 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8"] Oct 31 13:15:05.436307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.436292 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv"] Oct 31 13:15:05.452722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.416554 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 13:15:05.454317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.454285 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 13:15:05.532429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.523927 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv"] Oct 31 13:15:05.532429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.523988 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m"] Oct 31 13:15:05.532429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.525423 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.553622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.553576 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c2c98b\"/\"routing-overlay\"" Oct 31 13:15:05.554874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.554850 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c2c98b\"/\"agc-cred-2b449978b1c0\"" Oct 31 13:15:05.555114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.555099 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c2c98b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c2c98b\"" Oct 31 13:15:05.555291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.555279 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c2c98b\"/\"kube-root-ca.crt\"" Oct 31 13:15:05.555457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.555446 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c2c98b\"/\"openshift-service-ca.crt\"" Oct 31 13:15:05.555578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.555567 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s"] Oct 31 13:15:05.672895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.670289 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.672895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.670377 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.672895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.670488 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzm8x\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-kube-api-access-tzm8x\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.672895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.670670 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.796331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.795367 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.796331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.795463 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.796331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.795587 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzm8x\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-kube-api-access-tzm8x\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.796331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.795797 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.800251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.799688 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.810678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.806318 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.810678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.807267 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.834545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.834219 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzm8x\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-kube-api-access-tzm8x\") pod \"payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:05.971661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.971430 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk"] Oct 31 13:15:05.992091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.991955 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d"] Oct 31 13:15:05.999466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:05.999431 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv"] Oct 31 13:15:06.001467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.001445 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4"] Oct 31 13:15:06.098176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.097572 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm"] Oct 31 13:15:06.129198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.129157 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t"] Oct 31 13:15:06.137084 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:06.137052 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095b3e51_cd04_44ef_9798_fe0e94c19bef.slice/crio-0ef61e3a3775f05b2730eddd39daa4b15c6b55ef59a4ac6f4ef8f338d6e42143 WatchSource:0}: Error finding container 0ef61e3a3775f05b2730eddd39daa4b15c6b55ef59a4ac6f4ef8f338d6e42143: Status 404 returned error can't find the container with id 0ef61e3a3775f05b2730eddd39daa4b15c6b55ef59a4ac6f4ef8f338d6e42143 Oct 31 13:15:06.152115 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:06.152086 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1acca3_b662_4b50_8de7_4f3dfa151f2f.slice/crio-3a7a9142cff28d8f91f63d19e1ceb63fce6d70bee0ba74d2cab9bec917812d5f WatchSource:0}: Error finding container 3a7a9142cff28d8f91f63d19e1ceb63fce6d70bee0ba74d2cab9bec917812d5f: Status 404 returned error can't find the container with id 3a7a9142cff28d8f91f63d19e1ceb63fce6d70bee0ba74d2cab9bec917812d5f Oct 31 13:15:06.186424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.186137 5074 patch_prober.go:29] interesting pod/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:15:06.186424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.186191 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 31 13:15:06.214918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.205019 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:06.236433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.236398 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt"] Oct 31 13:15:06.243104 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:06.243044 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f9576c_b31b_4cb0_84b9_32a6c471ab16.slice/crio-0cc84fea818346f6c71d99842e9ebde1cfd152e41f38b4bd6d25e623145384fc WatchSource:0}: Error finding container 0cc84fea818346f6c71d99842e9ebde1cfd152e41f38b4bd6d25e623145384fc: Status 404 returned error can't find the container with id 0cc84fea818346f6c71d99842e9ebde1cfd152e41f38b4bd6d25e623145384fc Oct 31 13:15:06.244585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.244473 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k"] Oct 31 13:15:06.244585 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:06.244470 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8fd503_9169_4d5a_aab0_799659a0be24.slice/crio-cf57855331b4503690815315d355f367eb932fe20fa0762942e80f90f61778d7 WatchSource:0}: Error finding container cf57855331b4503690815315d355f367eb932fe20fa0762942e80f90f61778d7: Status 404 returned error can't find the container with id cf57855331b4503690815315d355f367eb932fe20fa0762942e80f90f61778d7 Oct 31 13:15:06.308721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.308624 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerStarted","Data":"1f9b232c7f1c31b8d1b32f3527a12a81b1647d6a4fab470ace6b5d848a47c91a"} Oct 31 13:15:06.314068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.314040 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:15:06.318675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.317078 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:15:06.321005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.320676 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" event={"ID":"d6f9576c-b31b-4cb0-84b9-32a6c471ab16","Type":"ContainerStarted","Data":"0cc84fea818346f6c71d99842e9ebde1cfd152e41f38b4bd6d25e623145384fc"} Oct 31 13:15:06.321526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.321411 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:15:06.323116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.323084 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" event={"ID":"1aafcb56-25d6-4d5e-83f0-98bb476db5a4","Type":"ContainerStarted","Data":"51da2e9fc9e7dcd912f48abe5ebe00289af81228df54f256aceb29a3512cae2e"} Oct 31 13:15:06.325261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.325238 5074 generic.go:356] "Generic (PLEG): container finished" podID="316711a0-4531-404b-9e48-8c7824f93978" containerID="e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269" exitCode=0 Oct 31 13:15:06.325346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.325304 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" event={"ID":"316711a0-4531-404b-9e48-8c7824f93978","Type":"ContainerDied","Data":"e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269"} Oct 31 13:15:06.325708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.325628 5074 scope.go:122] "RemoveContainer" containerID="e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269" Oct 31 13:15:06.328242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.328211 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:06.328324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.328283 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:06.328324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.328316 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:06.329105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.329084 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" event={"ID":"af8fd503-9169-4d5a-aab0-799659a0be24","Type":"ContainerStarted","Data":"cf57855331b4503690815315d355f367eb932fe20fa0762942e80f90f61778d7"} Oct 31 13:15:06.336022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.336001 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerStarted","Data":"3a7a9142cff28d8f91f63d19e1ceb63fce6d70bee0ba74d2cab9bec917812d5f"} Oct 31 13:15:06.339533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.339485 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" podStartSLOduration=6.339470122 podStartE2EDuration="6.339470122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:06.331275395 +0000 UTC m=+2600.508993382" watchObservedRunningTime="2026-10-31 13:15:06.339470122 +0000 UTC m=+2600.517188112" Oct 31 13:15:06.345779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.345755 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerStarted","Data":"a48019a62d294ae63926b14caa5203ddb0e8201acecbd6a99dfb321ad2b84f4f"} Oct 31 13:15:06.348584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.348551 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" event={"ID":"6a434213-99ab-4ceb-8a6e-424927391e99","Type":"ContainerStarted","Data":"a58f52e881007f3be9c1a6247a9050c3a242156e4b402a604273aa0938d459b6"} Oct 31 13:15:06.349380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.349357 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96"] Oct 31 13:15:06.350529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.350501 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" event={"ID":"095b3e51-cd04-44ef-9798-fe0e94c19bef","Type":"ContainerStarted","Data":"0ef61e3a3775f05b2730eddd39daa4b15c6b55ef59a4ac6f4ef8f338d6e42143"} Oct 31 13:15:06.353427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.353387 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" event={"ID":"2af9234f-cf40-4883-8d91-c392be69d425","Type":"ContainerStarted","Data":"6f4533b7bb9a97931d6e1a4c47a849823c8189d80df2eced51eacbb5ed51f81b"} Oct 31 13:15:06.355225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.355200 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" event={"ID":"03a45b88-fafa-420f-a7ff-99abed7bd013","Type":"ContainerStarted","Data":"cdc882db3b0a8bcfe0a03d6e35db6c1b05bae17beed553a0f9eb5e40c5708249"} Oct 31 13:15:06.356897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.356874 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" event={"ID":"d7399962-c419-4d22-85f0-6364c37e8379","Type":"ContainerStarted","Data":"1a85e7158cc627aaadfc08226032ad1908d799d865d90422237a8bae4eaf0774"} Oct 31 13:15:06.358612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.358589 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerStarted","Data":"6156ba012bfb7333255ffedd7ab1273368f85e4079d5388e3fc8036a01044102"} Oct 31 13:15:06.359851 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.359825 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm"] Oct 31 13:15:06.429138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.428963 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:15:06.430303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.430280 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:15:06.439274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.439249 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:15:06.471971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.471898 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" podStartSLOduration=6.471882183 podStartE2EDuration="6.471882183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:06.425236993 +0000 UTC m=+2600.602954978" watchObservedRunningTime="2026-10-31 13:15:06.471882183 +0000 UTC m=+2600.649600168" Oct 31 13:15:06.477075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.477044 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj"] Oct 31 13:15:06.504893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.504862 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp"] Oct 31 13:15:06.514254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.514229 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4"] Oct 31 13:15:06.839490 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:06.839456 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a434213_99ab_4ceb_8a6e_424927391e99.slice/crio-conmon-a58f52e881007f3be9c1a6247a9050c3a242156e4b402a604273aa0938d459b6.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:15:06.884720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:06.882312 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv"] Oct 31 13:15:07.404757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.404720 5074 generic.go:356] "Generic (PLEG): container finished" podID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerID="1f9b232c7f1c31b8d1b32f3527a12a81b1647d6a4fab470ace6b5d848a47c91a" exitCode=60 Oct 31 13:15:07.405101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.405068 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerDied","Data":"1f9b232c7f1c31b8d1b32f3527a12a81b1647d6a4fab470ace6b5d848a47c91a"} Oct 31 13:15:07.407701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.405680 5074 scope.go:122] "RemoveContainer" containerID="1f9b232c7f1c31b8d1b32f3527a12a81b1647d6a4fab470ace6b5d848a47c91a" Oct 31 13:15:07.416117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.416078 5074 generic.go:356] "Generic (PLEG): container finished" podID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerID="2c2d042290b3e5fc2aa96b742aa50462deb297ba48d47467dcb2aee5549cd307" exitCode=60 Oct 31 13:15:07.416411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.416377 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" event={"ID":"1aafcb56-25d6-4d5e-83f0-98bb476db5a4","Type":"ContainerDied","Data":"2c2d042290b3e5fc2aa96b742aa50462deb297ba48d47467dcb2aee5549cd307"} Oct 31 13:15:07.427439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.427398 5074 scope.go:122] "RemoveContainer" containerID="2c2d042290b3e5fc2aa96b742aa50462deb297ba48d47467dcb2aee5549cd307" Oct 31 13:15:07.429212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.429167 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" event={"ID":"316711a0-4531-404b-9e48-8c7824f93978","Type":"ContainerStarted","Data":"c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47"} Oct 31 13:15:07.439566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.439112 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" event={"ID":"c8c6d080-831c-498d-9d5c-6fa4c94870e5","Type":"ContainerStarted","Data":"f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3"} Oct 31 13:15:07.439776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.439752 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" event={"ID":"c8c6d080-831c-498d-9d5c-6fa4c94870e5","Type":"ContainerStarted","Data":"e6bf5a653ffce6f860ccaa29ad33174cf5f4ffce3e57c42e5fefbc52ec123e4e"} Oct 31 13:15:07.440023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.439998 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:07.445202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.444698 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerStarted","Data":"04b0a72b02bd8606c89b3e5016e4cc1f1b18084dc5d4c81fd4f953dfd2157983"} Oct 31 13:15:07.448724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.448136 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerStarted","Data":"c76e23488ffad3e32d77724ffe1c434f170dbe0995767f28b4cf509a104a7182"} Oct 31 13:15:07.454043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.454022 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" event={"ID":"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80","Type":"ContainerStarted","Data":"db2dfcf7097558c4651a16d1235a7b16b337b9713d0eaa69ceb234b0c732d9fa"} Oct 31 13:15:07.455931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.455905 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerStarted","Data":"49d42e2184746d0b8517bc4a1f55ad3adcc0131a9ba836d8bee21506abe392fb"} Oct 31 13:15:07.456101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.456081 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerStarted","Data":"afacf9ac07c63a3ab75b09d6604766c9872a8676b1c0fb383da72b016a0633ee"} Oct 31 13:15:07.458411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.458392 5074 generic.go:356] "Generic (PLEG): container finished" podID="6a434213-99ab-4ceb-8a6e-424927391e99" containerID="a58f52e881007f3be9c1a6247a9050c3a242156e4b402a604273aa0938d459b6" exitCode=60 Oct 31 13:15:07.459293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.459273 5074 scope.go:122] "RemoveContainer" containerID="a58f52e881007f3be9c1a6247a9050c3a242156e4b402a604273aa0938d459b6" Oct 31 13:15:07.459397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.459059 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" event={"ID":"6a434213-99ab-4ceb-8a6e-424927391e99","Type":"ContainerDied","Data":"a58f52e881007f3be9c1a6247a9050c3a242156e4b402a604273aa0938d459b6"} Oct 31 13:15:07.461073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.460830 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" event={"ID":"721a5252-6907-4590-a6fc-bd7e1dc2a780","Type":"ContainerStarted","Data":"7964efad0ea6f9764cf729bc72b36518af9ed54359630c4c1c1908a834f7889b"} Oct 31 13:15:07.463730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.462621 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" event={"ID":"095b3e51-cd04-44ef-9798-fe0e94c19bef","Type":"ContainerStarted","Data":"78dc8596f660bfec87703e032884cd49c4db7959c545d9a2168564a3ccb5833a"} Oct 31 13:15:07.465857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.464227 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" event={"ID":"2af9234f-cf40-4883-8d91-c392be69d425","Type":"ContainerStarted","Data":"270a0e510fb11944d7aeb4e321df180c0ab3c733755243d2b135d1779356b9c0"} Oct 31 13:15:07.466449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.466135 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" event={"ID":"28505a68-5e10-4461-a95e-5680564eb5b7","Type":"ContainerStarted","Data":"85cd3499394e2477ec8d2c6b62520a68d50e0ea842f71feb92d843a241e6475f"} Oct 31 13:15:07.466449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.466181 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" event={"ID":"28505a68-5e10-4461-a95e-5680564eb5b7","Type":"ContainerStarted","Data":"b9b92d604fcfdd50c28869c966f675f52d356751f23d4d3d281de93bd3ec8b7d"} Oct 31 13:15:07.467521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.467488 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" event={"ID":"f659ac7d-4b1e-429b-8f44-d41fab077a70","Type":"ContainerStarted","Data":"fb4ff96dfebff5ad647d63cc487f743e39b5ef20592f34ff00dc20197b0b59e2"} Oct 31 13:15:07.478072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.478023 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" event={"ID":"03a45b88-fafa-420f-a7ff-99abed7bd013","Type":"ContainerStarted","Data":"ab88c4a833c401f12b187f810272cf6a21c1b5768e96f998e96c9364b3a3ec60"} Oct 31 13:15:07.479616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.479567 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" podStartSLOduration=7.479551996 podStartE2EDuration="7.479551996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.475101406 +0000 UTC m=+2601.652819392" watchObservedRunningTime="2026-10-31 13:15:07.479551996 +0000 UTC m=+2601.657269982" Oct 31 13:15:07.489009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.484378 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" event={"ID":"d7399962-c419-4d22-85f0-6364c37e8379","Type":"ContainerStarted","Data":"4f815b7215311d05416a3708754399e6915bab6f2413d49dea0114237988e3b8"} Oct 31 13:15:07.489009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.488884 5074 scope.go:122] "RemoveContainer" containerID="4f815b7215311d05416a3708754399e6915bab6f2413d49dea0114237988e3b8" Oct 31 13:15:07.495526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.495496 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerStarted","Data":"2422849154a6e212ddb0e1a2eee97329cb506340afef61627efd9129169d9054"} Oct 31 13:15:07.495769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.495746 5074 scope.go:122] "RemoveContainer" containerID="2422849154a6e212ddb0e1a2eee97329cb506340afef61627efd9129169d9054" Oct 31 13:15:07.552951 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.552574 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" podStartSLOduration=4.552553211 podStartE2EDuration="4.552553211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.529106523 +0000 UTC m=+2601.706824512" watchObservedRunningTime="2026-10-31 13:15:07.552553211 +0000 UTC m=+2601.730271301" Oct 31 13:15:07.565043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.564983 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" podStartSLOduration=7.5649594 podStartE2EDuration="7.5649594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.551163751 +0000 UTC m=+2601.728881736" watchObservedRunningTime="2026-10-31 13:15:07.5649594 +0000 UTC m=+2601.742677387" Oct 31 13:15:07.628265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.627980 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" podStartSLOduration=7.627964853 podStartE2EDuration="7.627964853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.579941807 +0000 UTC m=+2601.757659796" watchObservedRunningTime="2026-10-31 13:15:07.627964853 +0000 UTC m=+2601.805682836" Oct 31 13:15:07.634073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.633996 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" podStartSLOduration=7.633977169 podStartE2EDuration="7.633977169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.610623658 +0000 UTC m=+2601.788341643" watchObservedRunningTime="2026-10-31 13:15:07.633977169 +0000 UTC m=+2601.811695152" Oct 31 13:15:07.826905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.826872 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:15:07.829370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.829303 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" podStartSLOduration=7.829290122 podStartE2EDuration="7.829290122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.725511053 +0000 UTC m=+2601.903229065" watchObservedRunningTime="2026-10-31 13:15:07.829290122 +0000 UTC m=+2602.007008106" Oct 31 13:15:07.848207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.848150 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" podStartSLOduration=7.848132549 podStartE2EDuration="7.848132549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.801600582 +0000 UTC m=+2601.979318567" watchObservedRunningTime="2026-10-31 13:15:07.848132549 +0000 UTC m=+2602.025850534" Oct 31 13:15:07.857782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.857729 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" podStartSLOduration=7.857711041 podStartE2EDuration="7.857711041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:07.836365036 +0000 UTC m=+2602.014083021" watchObservedRunningTime="2026-10-31 13:15:07.857711041 +0000 UTC m=+2602.035429045" Oct 31 13:15:08.005063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.993275 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:15:08.005063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.993390 5074 patch_prober.go:29] interesting pod/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:15:08.005063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:07.993431 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 31 13:15:08.521464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.521431 5074 generic.go:356] "Generic (PLEG): container finished" podID="d7399962-c419-4d22-85f0-6364c37e8379" containerID="4f815b7215311d05416a3708754399e6915bab6f2413d49dea0114237988e3b8" exitCode=60 Oct 31 13:15:08.521620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.521503 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" event={"ID":"d7399962-c419-4d22-85f0-6364c37e8379","Type":"ContainerDied","Data":"4f815b7215311d05416a3708754399e6915bab6f2413d49dea0114237988e3b8"} Oct 31 13:15:08.523515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.523487 5074 generic.go:356] "Generic (PLEG): container finished" podID="dddae608-97bc-47b7-882d-710474720e95" containerID="2422849154a6e212ddb0e1a2eee97329cb506340afef61627efd9129169d9054" exitCode=60 Oct 31 13:15:08.523666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.523557 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerDied","Data":"2422849154a6e212ddb0e1a2eee97329cb506340afef61627efd9129169d9054"} Oct 31 13:15:08.525002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.524978 5074 generic.go:356] "Generic (PLEG): container finished" podID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerID="4399cbd55baff697114d274a8f833b7410ddb7a9d5a2c9329f19c6f75405aba7" exitCode=60 Oct 31 13:15:08.525106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.525048 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" event={"ID":"d6f9576c-b31b-4cb0-84b9-32a6c471ab16","Type":"ContainerDied","Data":"4399cbd55baff697114d274a8f833b7410ddb7a9d5a2c9329f19c6f75405aba7"} Oct 31 13:15:08.525333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.525317 5074 scope.go:122] "RemoveContainer" containerID="4399cbd55baff697114d274a8f833b7410ddb7a9d5a2c9329f19c6f75405aba7" Oct 31 13:15:08.526947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.526925 5074 generic.go:356] "Generic (PLEG): container finished" podID="af8fd503-9169-4d5a-aab0-799659a0be24" containerID="436450be7e3ecae776a1fade91cc6b5a65d380a044f8aec26dfdd1b0bd99c136" exitCode=60 Oct 31 13:15:08.527062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.527043 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" event={"ID":"af8fd503-9169-4d5a-aab0-799659a0be24","Type":"ContainerDied","Data":"436450be7e3ecae776a1fade91cc6b5a65d380a044f8aec26dfdd1b0bd99c136"} Oct 31 13:15:08.527328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.527311 5074 scope.go:122] "RemoveContainer" containerID="436450be7e3ecae776a1fade91cc6b5a65d380a044f8aec26dfdd1b0bd99c136" Oct 31 13:15:08.531620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.531596 5074 generic.go:356] "Generic (PLEG): container finished" podID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerID="04b0a72b02bd8606c89b3e5016e4cc1f1b18084dc5d4c81fd4f953dfd2157983" exitCode=60 Oct 31 13:15:08.531758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.531718 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerDied","Data":"04b0a72b02bd8606c89b3e5016e4cc1f1b18084dc5d4c81fd4f953dfd2157983"} Oct 31 13:15:08.531983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.531965 5074 scope.go:122] "RemoveContainer" containerID="04b0a72b02bd8606c89b3e5016e4cc1f1b18084dc5d4c81fd4f953dfd2157983" Oct 31 13:15:08.533789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.533770 5074 generic.go:356] "Generic (PLEG): container finished" podID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerID="c76e23488ffad3e32d77724ffe1c434f170dbe0995767f28b4cf509a104a7182" exitCode=60 Oct 31 13:15:08.533889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.533799 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerDied","Data":"c76e23488ffad3e32d77724ffe1c434f170dbe0995767f28b4cf509a104a7182"} Oct 31 13:15:08.534121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.534103 5074 scope.go:122] "RemoveContainer" containerID="c76e23488ffad3e32d77724ffe1c434f170dbe0995767f28b4cf509a104a7182" Oct 31 13:15:08.535442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.535426 5074 generic.go:356] "Generic (PLEG): container finished" podID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerID="533378c55b1c90317fdb0fafea341f9ff6165a67ed04f3ca0078aea02d783624" exitCode=60 Oct 31 13:15:08.535592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.535504 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" event={"ID":"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80","Type":"ContainerDied","Data":"533378c55b1c90317fdb0fafea341f9ff6165a67ed04f3ca0078aea02d783624"} Oct 31 13:15:08.535750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.535733 5074 scope.go:122] "RemoveContainer" containerID="533378c55b1c90317fdb0fafea341f9ff6165a67ed04f3ca0078aea02d783624" Oct 31 13:15:08.537308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.537293 5074 generic.go:356] "Generic (PLEG): container finished" podID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerID="49d42e2184746d0b8517bc4a1f55ad3adcc0131a9ba836d8bee21506abe392fb" exitCode=60 Oct 31 13:15:08.537605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.537573 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerDied","Data":"49d42e2184746d0b8517bc4a1f55ad3adcc0131a9ba836d8bee21506abe392fb"} Oct 31 13:15:08.537739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.537723 5074 scope.go:122] "RemoveContainer" containerID="49d42e2184746d0b8517bc4a1f55ad3adcc0131a9ba836d8bee21506abe392fb" Oct 31 13:15:08.539797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.539782 5074 generic.go:356] "Generic (PLEG): container finished" podID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerID="5ab68ba666ae0c24bcdd8c5ca77484280c01e55ff4f9109356b7ff2b74693333" exitCode=60 Oct 31 13:15:08.539981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.539857 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" event={"ID":"721a5252-6907-4590-a6fc-bd7e1dc2a780","Type":"ContainerDied","Data":"5ab68ba666ae0c24bcdd8c5ca77484280c01e55ff4f9109356b7ff2b74693333"} Oct 31 13:15:08.540096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.540052 5074 scope.go:122] "RemoveContainer" containerID="5ab68ba666ae0c24bcdd8c5ca77484280c01e55ff4f9109356b7ff2b74693333" Oct 31 13:15:08.541604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.541583 5074 generic.go:356] "Generic (PLEG): container finished" podID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerID="78dc8596f660bfec87703e032884cd49c4db7959c545d9a2168564a3ccb5833a" exitCode=60 Oct 31 13:15:08.541757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.541677 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" event={"ID":"095b3e51-cd04-44ef-9798-fe0e94c19bef","Type":"ContainerDied","Data":"78dc8596f660bfec87703e032884cd49c4db7959c545d9a2168564a3ccb5833a"} Oct 31 13:15:08.542128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.542108 5074 scope.go:122] "RemoveContainer" containerID="78dc8596f660bfec87703e032884cd49c4db7959c545d9a2168564a3ccb5833a" Oct 31 13:15:08.543359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.543321 5074 generic.go:356] "Generic (PLEG): container finished" podID="2af9234f-cf40-4883-8d91-c392be69d425" containerID="270a0e510fb11944d7aeb4e321df180c0ab3c733755243d2b135d1779356b9c0" exitCode=60 Oct 31 13:15:08.543546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.543522 5074 scope.go:122] "RemoveContainer" containerID="270a0e510fb11944d7aeb4e321df180c0ab3c733755243d2b135d1779356b9c0" Oct 31 13:15:08.543703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.543483 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" event={"ID":"2af9234f-cf40-4883-8d91-c392be69d425","Type":"ContainerDied","Data":"270a0e510fb11944d7aeb4e321df180c0ab3c733755243d2b135d1779356b9c0"} Oct 31 13:15:08.547121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.547104 5074 generic.go:356] "Generic (PLEG): container finished" podID="28505a68-5e10-4461-a95e-5680564eb5b7" containerID="85cd3499394e2477ec8d2c6b62520a68d50e0ea842f71feb92d843a241e6475f" exitCode=60 Oct 31 13:15:08.547234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.547182 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" event={"ID":"28505a68-5e10-4461-a95e-5680564eb5b7","Type":"ContainerDied","Data":"85cd3499394e2477ec8d2c6b62520a68d50e0ea842f71feb92d843a241e6475f"} Oct 31 13:15:08.547940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.547922 5074 scope.go:122] "RemoveContainer" containerID="85cd3499394e2477ec8d2c6b62520a68d50e0ea842f71feb92d843a241e6475f" Oct 31 13:15:08.568305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.568273 5074 generic.go:356] "Generic (PLEG): container finished" podID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerID="967d7f9c094e68b1fb6c66f8feaae9586c46ef00e13829b2717f0cb4fbddb4b9" exitCode=60 Oct 31 13:15:08.568522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.568498 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" event={"ID":"f659ac7d-4b1e-429b-8f44-d41fab077a70","Type":"ContainerDied","Data":"967d7f9c094e68b1fb6c66f8feaae9586c46ef00e13829b2717f0cb4fbddb4b9"} Oct 31 13:15:08.568970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.568947 5074 scope.go:122] "RemoveContainer" containerID="967d7f9c094e68b1fb6c66f8feaae9586c46ef00e13829b2717f0cb4fbddb4b9" Oct 31 13:15:08.616072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.613588 5074 generic.go:356] "Generic (PLEG): container finished" podID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerID="ab88c4a833c401f12b187f810272cf6a21c1b5768e96f998e96c9364b3a3ec60" exitCode=60 Oct 31 13:15:08.616072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.613814 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" event={"ID":"03a45b88-fafa-420f-a7ff-99abed7bd013","Type":"ContainerDied","Data":"ab88c4a833c401f12b187f810272cf6a21c1b5768e96f998e96c9364b3a3ec60"} Oct 31 13:15:08.616072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.614154 5074 scope.go:122] "RemoveContainer" containerID="ab88c4a833c401f12b187f810272cf6a21c1b5768e96f998e96c9364b3a3ec60" Oct 31 13:15:08.993518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.993108 5074 patch_prober.go:29] interesting pod/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:15:08.993518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:08.993159 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 31 13:15:09.635580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.635547 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" event={"ID":"af8fd503-9169-4d5a-aab0-799659a0be24","Type":"ContainerStarted","Data":"03f6968a311a9de05fe2bb5a1d417e94f0c5c2b500add6418481bf3782731573"} Oct 31 13:15:09.639007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.638976 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerStarted","Data":"760a21577064783e288b8c07e765e513b3f9ac542aee5f4ff64b619623f0d2b8"} Oct 31 13:15:09.642714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.642348 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" event={"ID":"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80","Type":"ContainerStarted","Data":"7427610d0f14747d64ac0525ecb78ab5362423215c0dc5cb3c8f7ecea701d5ff"} Oct 31 13:15:09.644684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.644661 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerStarted","Data":"e9abfacc2c7078177a9ac14a549f52b28b42f014f3b8efaf1abccbd883db30e5"} Oct 31 13:15:09.646586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.646565 5074 generic.go:356] "Generic (PLEG): container finished" podID="6a434213-99ab-4ceb-8a6e-424927391e99" containerID="d1c100bd638640d9e39e9e0e334889be0783546c7d25766d12bf4519234153fc" exitCode=60 Oct 31 13:15:09.646716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.646642 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" event={"ID":"6a434213-99ab-4ceb-8a6e-424927391e99","Type":"ContainerDied","Data":"d1c100bd638640d9e39e9e0e334889be0783546c7d25766d12bf4519234153fc"} Oct 31 13:15:09.646716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.646696 5074 scope.go:122] "RemoveContainer" containerID="a58f52e881007f3be9c1a6247a9050c3a242156e4b402a604273aa0938d459b6" Oct 31 13:15:09.646866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.646849 5074 scope.go:122] "RemoveContainer" containerID="d1c100bd638640d9e39e9e0e334889be0783546c7d25766d12bf4519234153fc" Oct 31 13:15:09.647100 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:09.647079 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46_odh-ai-gateway-infra(6a434213-99ab-4ceb-8a6e-424927391e99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" Oct 31 13:15:09.649249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.649216 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" event={"ID":"721a5252-6907-4590-a6fc-bd7e1dc2a780","Type":"ContainerStarted","Data":"04bdb38c7fda94f4ece7bf3829ea890ae81ee43473ae98064e4781d114f23d0a"} Oct 31 13:15:09.651950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.651080 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" event={"ID":"095b3e51-cd04-44ef-9798-fe0e94c19bef","Type":"ContainerStarted","Data":"08c294588b733e5f29a489702da27b1fe9edb2c4cd538bab71efdd4c12b0815b"} Oct 31 13:15:09.655284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.655261 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" event={"ID":"d7399962-c419-4d22-85f0-6364c37e8379","Type":"ContainerStarted","Data":"c55f9cf51421c3ed94e92a2c674e0c4ed65632167dc9faffd98371b0d8edd780"} Oct 31 13:15:09.656983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.656963 5074 scope.go:122] "RemoveContainer" containerID="c55f9cf51421c3ed94e92a2c674e0c4ed65632167dc9faffd98371b0d8edd780" Oct 31 13:15:09.657039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.657017 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerStarted","Data":"3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c"} Oct 31 13:15:09.657285 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:09.657261 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv_odh-ai-gateway-infra(d7399962-c419-4d22-85f0-6364c37e8379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" podUID="d7399962-c419-4d22-85f0-6364c37e8379" Oct 31 13:15:09.657355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.657343 5074 scope.go:122] "RemoveContainer" containerID="3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c" Oct 31 13:15:09.657549 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:09.657532 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m_odh-ai-gateway-infra(dddae608-97bc-47b7-882d-710474720e95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" podUID="dddae608-97bc-47b7-882d-710474720e95" Oct 31 13:15:09.660149 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.659545 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerStarted","Data":"2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1"} Oct 31 13:15:09.661123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.661105 5074 scope.go:122] "RemoveContainer" containerID="2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1" Oct 31 13:15:09.661388 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:09.661364 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8_odh-ai-gateway-infra(1abcd738-c940-44ba-9b90-6081d4a6b861)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" Oct 31 13:15:09.662897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.662261 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" event={"ID":"d6f9576c-b31b-4cb0-84b9-32a6c471ab16","Type":"ContainerStarted","Data":"4145d446eb452dac06ed3639ab5c2bf1981ac386e087db1e57a14ecc5049634c"} Oct 31 13:15:09.665478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.664953 5074 generic.go:356] "Generic (PLEG): container finished" podID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerID="b604deae64fba3158f763051ff7caca823f0d8f7650b6c3e4250688c5e7fc4dc" exitCode=60 Oct 31 13:15:09.665478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.665020 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" event={"ID":"1aafcb56-25d6-4d5e-83f0-98bb476db5a4","Type":"ContainerDied","Data":"b604deae64fba3158f763051ff7caca823f0d8f7650b6c3e4250688c5e7fc4dc"} Oct 31 13:15:09.665478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.665427 5074 scope.go:122] "RemoveContainer" containerID="b604deae64fba3158f763051ff7caca823f0d8f7650b6c3e4250688c5e7fc4dc" Oct 31 13:15:09.665694 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:09.665676 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s_odh-ai-gateway-infra(1aafcb56-25d6-4d5e-83f0-98bb476db5a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" Oct 31 13:15:09.666261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.666217 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" podStartSLOduration=9.666202421 podStartE2EDuration="9.666202421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:09.660127291 +0000 UTC m=+2603.837845278" watchObservedRunningTime="2026-10-31 13:15:09.666202421 +0000 UTC m=+2603.843920410" Oct 31 13:15:09.729878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.729816 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" podStartSLOduration=9.729798128 podStartE2EDuration="9.729798128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:09.724221895 +0000 UTC m=+2603.901939890" watchObservedRunningTime="2026-10-31 13:15:09.729798128 +0000 UTC m=+2603.907516114" Oct 31 13:15:09.743127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.743079 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" podStartSLOduration=9.74306955 podStartE2EDuration="9.74306955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:09.741653767 +0000 UTC m=+2603.919371754" watchObservedRunningTime="2026-10-31 13:15:09.74306955 +0000 UTC m=+2603.920787533" Oct 31 13:15:09.832711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.832688 5074 scope.go:122] "RemoveContainer" containerID="2c2d042290b3e5fc2aa96b742aa50462deb297ba48d47467dcb2aee5549cd307" Oct 31 13:15:09.996657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.996113 5074 patch_prober.go:29] interesting pod/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:15:09.996657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:09.996186 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 31 13:15:10.009653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.008934 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" podStartSLOduration=10.008915295 podStartE2EDuration="10.008915295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:10.007069839 +0000 UTC m=+2604.184787826" watchObservedRunningTime="2026-10-31 13:15:10.008915295 +0000 UTC m=+2604.186633279" Oct 31 13:15:10.709944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.709907 5074 generic.go:356] "Generic (PLEG): container finished" podID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerID="2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1" exitCode=60 Oct 31 13:15:10.710123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.710006 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerDied","Data":"2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1"} Oct 31 13:15:10.710123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.710064 5074 scope.go:122] "RemoveContainer" containerID="1f9b232c7f1c31b8d1b32f3527a12a81b1647d6a4fab470ace6b5d848a47c91a" Oct 31 13:15:10.710558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.710536 5074 scope.go:122] "RemoveContainer" containerID="2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1" Oct 31 13:15:10.711172 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.711027 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8_odh-ai-gateway-infra(1abcd738-c940-44ba-9b90-6081d4a6b861)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" Oct 31 13:15:10.714129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.714108 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" event={"ID":"d6f9576c-b31b-4cb0-84b9-32a6c471ab16","Type":"ContainerDied","Data":"4145d446eb452dac06ed3639ab5c2bf1981ac386e087db1e57a14ecc5049634c"} Oct 31 13:15:10.714259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.714068 5074 generic.go:356] "Generic (PLEG): container finished" podID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerID="4145d446eb452dac06ed3639ab5c2bf1981ac386e087db1e57a14ecc5049634c" exitCode=60 Oct 31 13:15:10.714498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.714478 5074 scope.go:122] "RemoveContainer" containerID="4145d446eb452dac06ed3639ab5c2bf1981ac386e087db1e57a14ecc5049634c" Oct 31 13:15:10.714939 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.714913 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt_odh-ai-gateway-infra(d6f9576c-b31b-4cb0-84b9-32a6c471ab16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" Oct 31 13:15:10.717047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.717026 5074 scope.go:122] "RemoveContainer" containerID="b604deae64fba3158f763051ff7caca823f0d8f7650b6c3e4250688c5e7fc4dc" Oct 31 13:15:10.717541 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.717513 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s_odh-ai-gateway-infra(1aafcb56-25d6-4d5e-83f0-98bb476db5a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" Oct 31 13:15:10.719555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.719529 5074 generic.go:356] "Generic (PLEG): container finished" podID="af8fd503-9169-4d5a-aab0-799659a0be24" containerID="03f6968a311a9de05fe2bb5a1d417e94f0c5c2b500add6418481bf3782731573" exitCode=60 Oct 31 13:15:10.719700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.719681 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" event={"ID":"af8fd503-9169-4d5a-aab0-799659a0be24","Type":"ContainerDied","Data":"03f6968a311a9de05fe2bb5a1d417e94f0c5c2b500add6418481bf3782731573"} Oct 31 13:15:10.719990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.719971 5074 scope.go:122] "RemoveContainer" containerID="03f6968a311a9de05fe2bb5a1d417e94f0c5c2b500add6418481bf3782731573" Oct 31 13:15:10.720352 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.720330 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k_odh-ai-gateway-infra(af8fd503-9169-4d5a-aab0-799659a0be24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" Oct 31 13:15:10.722032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.722013 5074 generic.go:356] "Generic (PLEG): container finished" podID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerID="760a21577064783e288b8c07e765e513b3f9ac542aee5f4ff64b619623f0d2b8" exitCode=60 Oct 31 13:15:10.722119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.722094 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerDied","Data":"760a21577064783e288b8c07e765e513b3f9ac542aee5f4ff64b619623f0d2b8"} Oct 31 13:15:10.722521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.722503 5074 scope.go:122] "RemoveContainer" containerID="760a21577064783e288b8c07e765e513b3f9ac542aee5f4ff64b619623f0d2b8" Oct 31 13:15:10.723072 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.723050 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t_odh-ai-gateway-infra(2c1acca3-b662-4b50-8de7-4f3dfa151f2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" Oct 31 13:15:10.725095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.725073 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerStarted","Data":"31b58a3cf85820d74c87e0274f0ff905992969142f69c576cc8d398f65d7eb62"} Oct 31 13:15:10.726205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.726191 5074 scope.go:122] "RemoveContainer" containerID="4399cbd55baff697114d274a8f833b7410ddb7a9d5a2c9329f19c6f75405aba7" Oct 31 13:15:10.728218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.727598 5074 generic.go:356] "Generic (PLEG): container finished" podID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerID="7427610d0f14747d64ac0525ecb78ab5362423215c0dc5cb3c8f7ecea701d5ff" exitCode=60 Oct 31 13:15:10.728218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.727838 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" event={"ID":"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80","Type":"ContainerDied","Data":"7427610d0f14747d64ac0525ecb78ab5362423215c0dc5cb3c8f7ecea701d5ff"} Oct 31 13:15:10.728218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.727971 5074 scope.go:122] "RemoveContainer" containerID="7427610d0f14747d64ac0525ecb78ab5362423215c0dc5cb3c8f7ecea701d5ff" Oct 31 13:15:10.728432 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.728301 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4_odh-ai-gateway-infra(e1ac994b-957d-47e5-8c0d-20dbc4c2ea80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" Oct 31 13:15:10.729850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.729829 5074 generic.go:356] "Generic (PLEG): container finished" podID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerID="e9abfacc2c7078177a9ac14a549f52b28b42f014f3b8efaf1abccbd883db30e5" exitCode=60 Oct 31 13:15:10.730087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.730068 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerDied","Data":"e9abfacc2c7078177a9ac14a549f52b28b42f014f3b8efaf1abccbd883db30e5"} Oct 31 13:15:10.730243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.730197 5074 scope.go:122] "RemoveContainer" containerID="e9abfacc2c7078177a9ac14a549f52b28b42f014f3b8efaf1abccbd883db30e5" Oct 31 13:15:10.730500 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.730479 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96_odh-ai-gateway-infra(56c5cc9f-9c47-4155-855f-e8b9ad1a876e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" Oct 31 13:15:10.732248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.732183 5074 scope.go:122] "RemoveContainer" containerID="d1c100bd638640d9e39e9e0e334889be0783546c7d25766d12bf4519234153fc" Oct 31 13:15:10.732479 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.732452 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46_odh-ai-gateway-infra(6a434213-99ab-4ceb-8a6e-424927391e99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" Oct 31 13:15:10.736700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.736610 5074 generic.go:356] "Generic (PLEG): container finished" podID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerID="04bdb38c7fda94f4ece7bf3829ea890ae81ee43473ae98064e4781d114f23d0a" exitCode=60 Oct 31 13:15:10.736869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.736753 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" event={"ID":"721a5252-6907-4590-a6fc-bd7e1dc2a780","Type":"ContainerDied","Data":"04bdb38c7fda94f4ece7bf3829ea890ae81ee43473ae98064e4781d114f23d0a"} Oct 31 13:15:10.736947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.736930 5074 scope.go:122] "RemoveContainer" containerID="04bdb38c7fda94f4ece7bf3829ea890ae81ee43473ae98064e4781d114f23d0a" Oct 31 13:15:10.737187 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.737163 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj_odh-ai-gateway-infra(721a5252-6907-4590-a6fc-bd7e1dc2a780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" Oct 31 13:15:10.739024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.738993 5074 generic.go:356] "Generic (PLEG): container finished" podID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerID="08c294588b733e5f29a489702da27b1fe9edb2c4cd538bab71efdd4c12b0815b" exitCode=60 Oct 31 13:15:10.739113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.739070 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" event={"ID":"095b3e51-cd04-44ef-9798-fe0e94c19bef","Type":"ContainerDied","Data":"08c294588b733e5f29a489702da27b1fe9edb2c4cd538bab71efdd4c12b0815b"} Oct 31 13:15:10.739337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.739320 5074 scope.go:122] "RemoveContainer" containerID="08c294588b733e5f29a489702da27b1fe9edb2c4cd538bab71efdd4c12b0815b" Oct 31 13:15:10.739522 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.739505 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm_odh-ai-gateway-infra(095b3e51-cd04-44ef-9798-fe0e94c19bef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" Oct 31 13:15:10.741007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.740986 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" event={"ID":"2af9234f-cf40-4883-8d91-c392be69d425","Type":"ContainerStarted","Data":"c16ee75180b71119049818322127ee5b0fafc095a70c96dbeb0a52dea8809158"} Oct 31 13:15:10.743526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.743510 5074 scope.go:122] "RemoveContainer" containerID="436450be7e3ecae776a1fade91cc6b5a65d380a044f8aec26dfdd1b0bd99c136" Oct 31 13:15:10.745544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.745520 5074 generic.go:356] "Generic (PLEG): container finished" podID="28505a68-5e10-4461-a95e-5680564eb5b7" containerID="1b67ccd202545989d9ffadd91b6738958a80e99ff6fe48cb84d3211a94be1e77" exitCode=60 Oct 31 13:15:10.745648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.745619 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" event={"ID":"28505a68-5e10-4461-a95e-5680564eb5b7","Type":"ContainerDied","Data":"1b67ccd202545989d9ffadd91b6738958a80e99ff6fe48cb84d3211a94be1e77"} Oct 31 13:15:10.746010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.745978 5074 scope.go:122] "RemoveContainer" containerID="1b67ccd202545989d9ffadd91b6738958a80e99ff6fe48cb84d3211a94be1e77" Oct 31 13:15:10.746295 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.746258 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm_odh-ai-gateway-infra(28505a68-5e10-4461-a95e-5680564eb5b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" Oct 31 13:15:10.747970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.747794 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" event={"ID":"f659ac7d-4b1e-429b-8f44-d41fab077a70","Type":"ContainerStarted","Data":"4ca37820c350da71a859b56b7036bc55c6c80c1ec3f5cb2182a75f580d041f5a"} Oct 31 13:15:10.749913 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.749838 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" event={"ID":"03a45b88-fafa-420f-a7ff-99abed7bd013","Type":"ContainerStarted","Data":"52c83985aa622aa988f6930473c778db492592aa13ad432d0bf61841b6d613e7"} Oct 31 13:15:10.752024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.752003 5074 generic.go:356] "Generic (PLEG): container finished" podID="d7399962-c419-4d22-85f0-6364c37e8379" containerID="c55f9cf51421c3ed94e92a2c674e0c4ed65632167dc9faffd98371b0d8edd780" exitCode=60 Oct 31 13:15:10.752122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.752059 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" event={"ID":"d7399962-c419-4d22-85f0-6364c37e8379","Type":"ContainerDied","Data":"c55f9cf51421c3ed94e92a2c674e0c4ed65632167dc9faffd98371b0d8edd780"} Oct 31 13:15:10.752402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.752345 5074 scope.go:122] "RemoveContainer" containerID="c55f9cf51421c3ed94e92a2c674e0c4ed65632167dc9faffd98371b0d8edd780" Oct 31 13:15:10.752816 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.752787 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv_odh-ai-gateway-infra(d7399962-c419-4d22-85f0-6364c37e8379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" podUID="d7399962-c419-4d22-85f0-6364c37e8379" Oct 31 13:15:10.754747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.754724 5074 generic.go:356] "Generic (PLEG): container finished" podID="dddae608-97bc-47b7-882d-710474720e95" containerID="3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c" exitCode=60 Oct 31 13:15:10.754840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.754761 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerDied","Data":"3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c"} Oct 31 13:15:10.755040 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.755025 5074 scope.go:122] "RemoveContainer" containerID="3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c" Oct 31 13:15:10.755250 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:10.755232 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m_odh-ai-gateway-infra(dddae608-97bc-47b7-882d-710474720e95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" podUID="dddae608-97bc-47b7-882d-710474720e95" Oct 31 13:15:10.762709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.762691 5074 scope.go:122] "RemoveContainer" containerID="04b0a72b02bd8606c89b3e5016e4cc1f1b18084dc5d4c81fd4f953dfd2157983" Oct 31 13:15:10.809962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.809937 5074 scope.go:122] "RemoveContainer" containerID="533378c55b1c90317fdb0fafea341f9ff6165a67ed04f3ca0078aea02d783624" Oct 31 13:15:10.858070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.858046 5074 scope.go:122] "RemoveContainer" containerID="49d42e2184746d0b8517bc4a1f55ad3adcc0131a9ba836d8bee21506abe392fb" Oct 31 13:15:10.915982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.915962 5074 scope.go:122] "RemoveContainer" containerID="5ab68ba666ae0c24bcdd8c5ca77484280c01e55ff4f9109356b7ff2b74693333" Oct 31 13:15:10.998725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.996834 5074 patch_prober.go:29] interesting pod/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:15:10.998725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:10.996892 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 31 13:15:11.016216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.016168 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" podStartSLOduration=11.016155076 podStartE2EDuration="11.016155076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:11.011921544 +0000 UTC m=+2605.189639531" watchObservedRunningTime="2026-10-31 13:15:11.016155076 +0000 UTC m=+2605.193873087" Oct 31 13:15:11.034385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.034363 5074 scope.go:122] "RemoveContainer" containerID="78dc8596f660bfec87703e032884cd49c4db7959c545d9a2168564a3ccb5833a" Oct 31 13:15:11.051770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.051726 5074 scope.go:122] "RemoveContainer" containerID="85cd3499394e2477ec8d2c6b62520a68d50e0ea842f71feb92d843a241e6475f" Oct 31 13:15:11.066431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.066411 5074 scope.go:122] "RemoveContainer" containerID="4f815b7215311d05416a3708754399e6915bab6f2413d49dea0114237988e3b8" Oct 31 13:15:11.104571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.104546 5074 scope.go:122] "RemoveContainer" containerID="2422849154a6e212ddb0e1a2eee97329cb506340afef61627efd9129169d9054" Oct 31 13:15:11.779665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.779359 5074 scope.go:122] "RemoveContainer" containerID="04bdb38c7fda94f4ece7bf3829ea890ae81ee43473ae98064e4781d114f23d0a" Oct 31 13:15:11.780033 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.779998 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj_odh-ai-gateway-infra(721a5252-6907-4590-a6fc-bd7e1dc2a780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" Oct 31 13:15:11.795132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.795102 5074 scope.go:122] "RemoveContainer" containerID="08c294588b733e5f29a489702da27b1fe9edb2c4cd538bab71efdd4c12b0815b" Oct 31 13:15:11.806415 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.806377 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm_odh-ai-gateway-infra(095b3e51-cd04-44ef-9798-fe0e94c19bef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" Oct 31 13:15:11.807749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.807596 5074 generic.go:356] "Generic (PLEG): container finished" podID="2af9234f-cf40-4883-8d91-c392be69d425" containerID="c16ee75180b71119049818322127ee5b0fafc095a70c96dbeb0a52dea8809158" exitCode=60 Oct 31 13:15:11.808139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.808117 5074 scope.go:122] "RemoveContainer" containerID="c16ee75180b71119049818322127ee5b0fafc095a70c96dbeb0a52dea8809158" Oct 31 13:15:11.808250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.807615 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" event={"ID":"2af9234f-cf40-4883-8d91-c392be69d425","Type":"ContainerDied","Data":"c16ee75180b71119049818322127ee5b0fafc095a70c96dbeb0a52dea8809158"} Oct 31 13:15:11.808323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.808297 5074 scope.go:122] "RemoveContainer" containerID="270a0e510fb11944d7aeb4e321df180c0ab3c733755243d2b135d1779356b9c0" Oct 31 13:15:11.809337 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.809300 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4_odh-ai-gateway-infra(2af9234f-cf40-4883-8d91-c392be69d425)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" podUID="2af9234f-cf40-4883-8d91-c392be69d425" Oct 31 13:15:11.814155 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.814125 5074 generic.go:356] "Generic (PLEG): container finished" podID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerID="4ca37820c350da71a859b56b7036bc55c6c80c1ec3f5cb2182a75f580d041f5a" exitCode=60 Oct 31 13:15:11.814271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.814217 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" event={"ID":"f659ac7d-4b1e-429b-8f44-d41fab077a70","Type":"ContainerDied","Data":"4ca37820c350da71a859b56b7036bc55c6c80c1ec3f5cb2182a75f580d041f5a"} Oct 31 13:15:11.814569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.814548 5074 scope.go:122] "RemoveContainer" containerID="4ca37820c350da71a859b56b7036bc55c6c80c1ec3f5cb2182a75f580d041f5a" Oct 31 13:15:11.815111 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.815082 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp_odh-ai-gateway-infra(f659ac7d-4b1e-429b-8f44-d41fab077a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" Oct 31 13:15:11.816532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.816511 5074 generic.go:356] "Generic (PLEG): container finished" podID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerID="52c83985aa622aa988f6930473c778db492592aa13ad432d0bf61841b6d613e7" exitCode=60 Oct 31 13:15:11.816680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.816603 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" event={"ID":"03a45b88-fafa-420f-a7ff-99abed7bd013","Type":"ContainerDied","Data":"52c83985aa622aa988f6930473c778db492592aa13ad432d0bf61841b6d613e7"} Oct 31 13:15:11.816895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.816876 5074 scope.go:122] "RemoveContainer" containerID="52c83985aa622aa988f6930473c778db492592aa13ad432d0bf61841b6d613e7" Oct 31 13:15:11.817251 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.817225 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d_odh-ai-gateway-infra(03a45b88-fafa-420f-a7ff-99abed7bd013)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" Oct 31 13:15:11.819008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.818985 5074 scope.go:122] "RemoveContainer" containerID="c55f9cf51421c3ed94e92a2c674e0c4ed65632167dc9faffd98371b0d8edd780" Oct 31 13:15:11.819350 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.819326 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv_odh-ai-gateway-infra(d7399962-c419-4d22-85f0-6364c37e8379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" podUID="d7399962-c419-4d22-85f0-6364c37e8379" Oct 31 13:15:11.820793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.820775 5074 scope.go:122] "RemoveContainer" containerID="3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c" Oct 31 13:15:11.821078 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.821056 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m_odh-ai-gateway-infra(dddae608-97bc-47b7-882d-710474720e95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" podUID="dddae608-97bc-47b7-882d-710474720e95" Oct 31 13:15:11.822566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.822544 5074 scope.go:122] "RemoveContainer" containerID="2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1" Oct 31 13:15:11.822896 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.822872 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8_odh-ai-gateway-infra(1abcd738-c940-44ba-9b90-6081d4a6b861)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" Oct 31 13:15:11.824136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.824119 5074 scope.go:122] "RemoveContainer" containerID="4145d446eb452dac06ed3639ab5c2bf1981ac386e087db1e57a14ecc5049634c" Oct 31 13:15:11.824403 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.824371 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt_odh-ai-gateway-infra(d6f9576c-b31b-4cb0-84b9-32a6c471ab16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" Oct 31 13:15:11.825756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.825738 5074 scope.go:122] "RemoveContainer" containerID="03f6968a311a9de05fe2bb5a1d417e94f0c5c2b500add6418481bf3782731573" Oct 31 13:15:11.826020 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.825997 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k_odh-ai-gateway-infra(af8fd503-9169-4d5a-aab0-799659a0be24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" Oct 31 13:15:11.826316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.826299 5074 scope.go:122] "RemoveContainer" containerID="967d7f9c094e68b1fb6c66f8feaae9586c46ef00e13829b2717f0cb4fbddb4b9" Oct 31 13:15:11.827299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.827278 5074 scope.go:122] "RemoveContainer" containerID="760a21577064783e288b8c07e765e513b3f9ac542aee5f4ff64b619623f0d2b8" Oct 31 13:15:11.827587 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.827562 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t_odh-ai-gateway-infra(2c1acca3-b662-4b50-8de7-4f3dfa151f2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" Oct 31 13:15:11.829874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.829851 5074 generic.go:356] "Generic (PLEG): container finished" podID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerID="31b58a3cf85820d74c87e0274f0ff905992969142f69c576cc8d398f65d7eb62" exitCode=60 Oct 31 13:15:11.830082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.829934 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerDied","Data":"31b58a3cf85820d74c87e0274f0ff905992969142f69c576cc8d398f65d7eb62"} Oct 31 13:15:11.830169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.830088 5074 scope.go:122] "RemoveContainer" containerID="31b58a3cf85820d74c87e0274f0ff905992969142f69c576cc8d398f65d7eb62" Oct 31 13:15:11.830673 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.830627 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk_odh-ai-gateway-infra(53dd0ac7-66bd-4841-aa0e-0a0b41294dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" Oct 31 13:15:11.832162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.832142 5074 scope.go:122] "RemoveContainer" containerID="7427610d0f14747d64ac0525ecb78ab5362423215c0dc5cb3c8f7ecea701d5ff" Oct 31 13:15:11.832354 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.832335 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4_odh-ai-gateway-infra(e1ac994b-957d-47e5-8c0d-20dbc4c2ea80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" Oct 31 13:15:11.834328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.834310 5074 scope.go:122] "RemoveContainer" containerID="e9abfacc2c7078177a9ac14a549f52b28b42f014f3b8efaf1abccbd883db30e5" Oct 31 13:15:11.834501 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:11.834483 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96_odh-ai-gateway-infra(56c5cc9f-9c47-4155-855f-e8b9ad1a876e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" Oct 31 13:15:11.842527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.842494 5074 scope.go:122] "RemoveContainer" containerID="ab88c4a833c401f12b187f810272cf6a21c1b5768e96f998e96c9364b3a3ec60" Oct 31 13:15:11.858245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.858221 5074 scope.go:122] "RemoveContainer" containerID="c76e23488ffad3e32d77724ffe1c434f170dbe0995767f28b4cf509a104a7182" Oct 31 13:15:11.997659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.996988 5074 patch_prober.go:29] interesting pod/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:15:11.997659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:11.997045 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 31 13:15:12.846166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:12.846133 5074 scope.go:122] "RemoveContainer" containerID="31b58a3cf85820d74c87e0274f0ff905992969142f69c576cc8d398f65d7eb62" Oct 31 13:15:12.846582 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:12.846561 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk_odh-ai-gateway-infra(53dd0ac7-66bd-4841-aa0e-0a0b41294dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" Oct 31 13:15:12.848000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:12.847975 5074 scope.go:122] "RemoveContainer" containerID="c16ee75180b71119049818322127ee5b0fafc095a70c96dbeb0a52dea8809158" Oct 31 13:15:12.848420 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:12.848396 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4_odh-ai-gateway-infra(2af9234f-cf40-4883-8d91-c392be69d425)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" podUID="2af9234f-cf40-4883-8d91-c392be69d425" Oct 31 13:15:12.849528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:12.849480 5074 scope.go:122] "RemoveContainer" containerID="4ca37820c350da71a859b56b7036bc55c6c80c1ec3f5cb2182a75f580d041f5a" Oct 31 13:15:12.849933 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:12.849902 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp_odh-ai-gateway-infra(f659ac7d-4b1e-429b-8f44-d41fab077a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" Oct 31 13:15:12.851143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:12.851123 5074 scope.go:122] "RemoveContainer" containerID="52c83985aa622aa988f6930473c778db492592aa13ad432d0bf61841b6d613e7" Oct 31 13:15:12.851379 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:12.851361 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d_odh-ai-gateway-infra(03a45b88-fafa-420f-a7ff-99abed7bd013)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" Oct 31 13:15:12.999657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:12.999086 5074 patch_prober.go:29] interesting pod/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 31 13:15:12.999657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:12.999156 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 31 13:15:13.994048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:13.994000 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:15:13.994527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:13.994416 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:15:13.995245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:13.995210 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:15:16.593174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:16.581326 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv"] Oct 31 13:15:17.193609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:17.193578 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:17.238575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:17.238538 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7399962-c419-4d22-85f0-6364c37e8379-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7399962-c419-4d22-85f0-6364c37e8379-maas-api-service-ca\") pod \"d7399962-c419-4d22-85f0-6364c37e8379\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " Oct 31 13:15:17.238839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:17.238818 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7399962-c419-4d22-85f0-6364c37e8379-kube-api-access-xjslw\" (UniqueName: \"kubernetes.io/projected/d7399962-c419-4d22-85f0-6364c37e8379-kube-api-access-xjslw\") pod \"d7399962-c419-4d22-85f0-6364c37e8379\" (UID: \"d7399962-c419-4d22-85f0-6364c37e8379\") " Oct 31 13:15:17.238976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:17.238951 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7399962-c419-4d22-85f0-6364c37e8379-maas-api-service-ca" pod "d7399962-c419-4d22-85f0-6364c37e8379" (UID: "d7399962-c419-4d22-85f0-6364c37e8379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:17.239814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:17.239754 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7399962-c419-4d22-85f0-6364c37e8379-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:17.241381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:17.241356 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7399962-c419-4d22-85f0-6364c37e8379-kube-api-access-xjslw" pod "d7399962-c419-4d22-85f0-6364c37e8379" (UID: "d7399962-c419-4d22-85f0-6364c37e8379"). InnerVolumeSpecName "kube-api-access-xjslw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:17.342060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:17.342006 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjslw\" (UniqueName: \"kubernetes.io/projected/d7399962-c419-4d22-85f0-6364c37e8379-kube-api-access-xjslw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:18.147333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:18.147304 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" Oct 31 13:15:18.147757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:18.147301 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv" event={"ID":"d7399962-c419-4d22-85f0-6364c37e8379","Type":"ContainerDied","Data":"1a85e7158cc627aaadfc08226032ad1908d799d865d90422237a8bae4eaf0774"} Oct 31 13:15:18.147757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:18.147436 5074 scope.go:122] "RemoveContainer" containerID="c55f9cf51421c3ed94e92a2c674e0c4ed65632167dc9faffd98371b0d8edd780" Oct 31 13:15:18.169692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:18.169618 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv"] Oct 31 13:15:18.175314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:18.175286 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-fa8c3f-29890875-qg8kv"] Oct 31 13:15:19.668714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:19.668673 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:15:19.830667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:19.830604 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7"] Oct 31 13:15:19.830872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:19.830844 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" containerID="cri-o://c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47" gracePeriod=30 Oct 31 13:15:20.153866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:20.153815 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7399962-c419-4d22-85f0-6364c37e8379" path="/var/lib/kubelet/pods/d7399962-c419-4d22-85f0-6364c37e8379/volumes" Oct 31 13:15:20.932288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:20.932117 5074 scope.go:122] "RemoveContainer" containerID="03f6968a311a9de05fe2bb5a1d417e94f0c5c2b500add6418481bf3782731573" Oct 31 13:15:20.932809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:20.932372 5074 scope.go:122] "RemoveContainer" containerID="b604deae64fba3158f763051ff7caca823f0d8f7650b6c3e4250688c5e7fc4dc" Oct 31 13:15:20.932809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:20.932461 5074 scope.go:122] "RemoveContainer" containerID="d1c100bd638640d9e39e9e0e334889be0783546c7d25766d12bf4519234153fc" Oct 31 13:15:21.931492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:21.931407 5074 scope.go:122] "RemoveContainer" containerID="760a21577064783e288b8c07e765e513b3f9ac542aee5f4ff64b619623f0d2b8" Oct 31 13:15:21.931683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:21.931584 5074 scope.go:122] "RemoveContainer" containerID="2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1" Oct 31 13:15:21.931772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:21.931752 5074 scope.go:122] "RemoveContainer" containerID="3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c" Oct 31 13:15:21.931932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:21.931912 5074 scope.go:122] "RemoveContainer" containerID="4ca37820c350da71a859b56b7036bc55c6c80c1ec3f5cb2182a75f580d041f5a" Oct 31 13:15:22.207918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.207826 5074 generic.go:356] "Generic (PLEG): container finished" podID="af8fd503-9169-4d5a-aab0-799659a0be24" containerID="47d07f9c2de3fe5cb6eb5c9eb8cd894f9203e9e69830bd9c84333174f31cf53d" exitCode=60 Oct 31 13:15:22.208526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.207895 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" event={"ID":"af8fd503-9169-4d5a-aab0-799659a0be24","Type":"ContainerDied","Data":"47d07f9c2de3fe5cb6eb5c9eb8cd894f9203e9e69830bd9c84333174f31cf53d"} Oct 31 13:15:22.208660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.208587 5074 scope.go:122] "RemoveContainer" containerID="03f6968a311a9de05fe2bb5a1d417e94f0c5c2b500add6418481bf3782731573" Oct 31 13:15:22.209213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.209156 5074 scope.go:122] "RemoveContainer" containerID="47d07f9c2de3fe5cb6eb5c9eb8cd894f9203e9e69830bd9c84333174f31cf53d" Oct 31 13:15:22.209678 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:22.209648 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k_odh-ai-gateway-infra(af8fd503-9169-4d5a-aab0-799659a0be24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" Oct 31 13:15:22.216310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.216042 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" event={"ID":"6a434213-99ab-4ceb-8a6e-424927391e99","Type":"ContainerStarted","Data":"234b0ac0c0fdc8b7678fbac4c52c19616daef8ccf55dead705a0bae6943eb5c1"} Oct 31 13:15:22.222280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.221472 5074 generic.go:356] "Generic (PLEG): container finished" podID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerID="7a63561992e9d4f8c55a9bcd0d68d0e4a09eb84ea80e0c130565c6753e27d2a5" exitCode=60 Oct 31 13:15:22.222280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.221514 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" event={"ID":"1aafcb56-25d6-4d5e-83f0-98bb476db5a4","Type":"ContainerDied","Data":"7a63561992e9d4f8c55a9bcd0d68d0e4a09eb84ea80e0c130565c6753e27d2a5"} Oct 31 13:15:22.222280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.221913 5074 scope.go:122] "RemoveContainer" containerID="7a63561992e9d4f8c55a9bcd0d68d0e4a09eb84ea80e0c130565c6753e27d2a5" Oct 31 13:15:22.222280 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:22.222241 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s_odh-ai-gateway-infra(1aafcb56-25d6-4d5e-83f0-98bb476db5a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" Oct 31 13:15:22.881507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.881478 5074 scope.go:122] "RemoveContainer" containerID="b604deae64fba3158f763051ff7caca823f0d8f7650b6c3e4250688c5e7fc4dc" Oct 31 13:15:22.936846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.936817 5074 scope.go:122] "RemoveContainer" containerID="31b58a3cf85820d74c87e0274f0ff905992969142f69c576cc8d398f65d7eb62" Oct 31 13:15:22.938871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.937975 5074 scope.go:122] "RemoveContainer" containerID="04bdb38c7fda94f4ece7bf3829ea890ae81ee43473ae98064e4781d114f23d0a" Oct 31 13:15:22.938871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.938068 5074 scope.go:122] "RemoveContainer" containerID="7427610d0f14747d64ac0525ecb78ab5362423215c0dc5cb3c8f7ecea701d5ff" Oct 31 13:15:22.938871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:22.938404 5074 scope.go:122] "RemoveContainer" containerID="1b67ccd202545989d9ffadd91b6738958a80e99ff6fe48cb84d3211a94be1e77" Oct 31 13:15:23.232312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.232277 5074 generic.go:356] "Generic (PLEG): container finished" podID="6a434213-99ab-4ceb-8a6e-424927391e99" containerID="234b0ac0c0fdc8b7678fbac4c52c19616daef8ccf55dead705a0bae6943eb5c1" exitCode=60 Oct 31 13:15:23.232797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.232344 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" event={"ID":"6a434213-99ab-4ceb-8a6e-424927391e99","Type":"ContainerDied","Data":"234b0ac0c0fdc8b7678fbac4c52c19616daef8ccf55dead705a0bae6943eb5c1"} Oct 31 13:15:23.232797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.232402 5074 scope.go:122] "RemoveContainer" containerID="d1c100bd638640d9e39e9e0e334889be0783546c7d25766d12bf4519234153fc" Oct 31 13:15:23.232878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.232851 5074 scope.go:122] "RemoveContainer" containerID="234b0ac0c0fdc8b7678fbac4c52c19616daef8ccf55dead705a0bae6943eb5c1" Oct 31 13:15:23.233292 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:23.233267 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46_odh-ai-gateway-infra(6a434213-99ab-4ceb-8a6e-424927391e99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" Oct 31 13:15:23.234836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.234811 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" event={"ID":"f659ac7d-4b1e-429b-8f44-d41fab077a70","Type":"ContainerStarted","Data":"5564161a91c1c508f024d994f323bfbf4eeabcf60170df8f0a9519f49c6cbffb"} Oct 31 13:15:23.236984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.236947 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerStarted","Data":"cfe66fb6b043d985c446ad7386d05f5d2e3c74e7f91ec695c3a746e3cccf1791"} Oct 31 13:15:23.239108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.239084 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerStarted","Data":"54ae67eff2732cc59e7ad9b3777ea92b63e7f55719ebe3c49976e6edb9254b1f"} Oct 31 13:15:23.244688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.244663 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerStarted","Data":"168baa1b15e59ec3108e5f48f196ea479f024aca734204d37970a7b81537d356"} Oct 31 13:15:23.305770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.303920 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k"] Oct 31 13:15:23.375545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.375454 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s"] Oct 31 13:15:23.397654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.397596 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46"] Oct 31 13:15:23.428672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.425910 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" podStartSLOduration=23.425894205 podStartE2EDuration="23.425894205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:23.386950152 +0000 UTC m=+2617.564668137" watchObservedRunningTime="2026-10-31 13:15:23.425894205 +0000 UTC m=+2617.603612189" Oct 31 13:15:23.800872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.800832 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:23.805379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.805341 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:23.928590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.928504 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af8fd503-9169-4d5a-aab0-799659a0be24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8fd503-9169-4d5a-aab0-799659a0be24-maas-api-service-ca\") pod \"af8fd503-9169-4d5a-aab0-799659a0be24\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " Oct 31 13:15:23.929212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.929185 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-maas-api-service-ca\") pod \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " Oct 31 13:15:23.929368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.929352 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-kube-api-access-8jwdc\" (UniqueName: \"kubernetes.io/projected/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-kube-api-access-8jwdc\") pod \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\" (UID: \"1aafcb56-25d6-4d5e-83f0-98bb476db5a4\") " Oct 31 13:15:23.929479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.929465 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8fd503-9169-4d5a-aab0-799659a0be24-kube-api-access-rlzb6\" (UniqueName: \"kubernetes.io/projected/af8fd503-9169-4d5a-aab0-799659a0be24-kube-api-access-rlzb6\") pod \"af8fd503-9169-4d5a-aab0-799659a0be24\" (UID: \"af8fd503-9169-4d5a-aab0-799659a0be24\") " Oct 31 13:15:23.937697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.932156 5074 scope.go:122] "RemoveContainer" containerID="e9abfacc2c7078177a9ac14a549f52b28b42f014f3b8efaf1abccbd883db30e5" Oct 31 13:15:23.937697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.935702 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-maas-api-service-ca" pod "1aafcb56-25d6-4d5e-83f0-98bb476db5a4" (UID: "1aafcb56-25d6-4d5e-83f0-98bb476db5a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:23.937697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.936091 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8fd503-9169-4d5a-aab0-799659a0be24-maas-api-service-ca" pod "af8fd503-9169-4d5a-aab0-799659a0be24" (UID: "af8fd503-9169-4d5a-aab0-799659a0be24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:23.940389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.938420 5074 scope.go:122] "RemoveContainer" containerID="08c294588b733e5f29a489702da27b1fe9edb2c4cd538bab71efdd4c12b0815b" Oct 31 13:15:23.940389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.939303 5074 scope.go:122] "RemoveContainer" containerID="4145d446eb452dac06ed3639ab5c2bf1981ac386e087db1e57a14ecc5049634c" Oct 31 13:15:23.941773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.941252 5074 scope.go:122] "RemoveContainer" containerID="c16ee75180b71119049818322127ee5b0fafc095a70c96dbeb0a52dea8809158" Oct 31 13:15:23.981817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.979864 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8fd503-9169-4d5a-aab0-799659a0be24-kube-api-access-rlzb6" pod "af8fd503-9169-4d5a-aab0-799659a0be24" (UID: "af8fd503-9169-4d5a-aab0-799659a0be24"). InnerVolumeSpecName "kube-api-access-rlzb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:23.981817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:23.979960 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-kube-api-access-8jwdc" pod "1aafcb56-25d6-4d5e-83f0-98bb476db5a4" (UID: "1aafcb56-25d6-4d5e-83f0-98bb476db5a4"). InnerVolumeSpecName "kube-api-access-8jwdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:24.032709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.032624 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:24.032709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.032675 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwdc\" (UniqueName: \"kubernetes.io/projected/1aafcb56-25d6-4d5e-83f0-98bb476db5a4-kube-api-access-8jwdc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:24.032709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.032692 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzb6\" (UniqueName: \"kubernetes.io/projected/af8fd503-9169-4d5a-aab0-799659a0be24-kube-api-access-rlzb6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:24.032709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.032708 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8fd503-9169-4d5a-aab0-799659a0be24-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:24.255543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.255515 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" Oct 31 13:15:24.255543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.255523 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s" event={"ID":"1aafcb56-25d6-4d5e-83f0-98bb476db5a4","Type":"ContainerDied","Data":"51da2e9fc9e7dcd912f48abe5ebe00289af81228df54f256aceb29a3512cae2e"} Oct 31 13:15:24.256128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.255580 5074 scope.go:122] "RemoveContainer" containerID="7a63561992e9d4f8c55a9bcd0d68d0e4a09eb84ea80e0c130565c6753e27d2a5" Oct 31 13:15:24.264317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.257122 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" event={"ID":"af8fd503-9169-4d5a-aab0-799659a0be24","Type":"ContainerDied","Data":"cf57855331b4503690815315d355f367eb932fe20fa0762942e80f90f61778d7"} Oct 31 13:15:24.264317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.257129 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k" Oct 31 13:15:24.272020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.272000 5074 scope.go:122] "RemoveContainer" containerID="47d07f9c2de3fe5cb6eb5c9eb8cd894f9203e9e69830bd9c84333174f31cf53d" Oct 31 13:15:24.295700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.288121 5074 generic.go:356] "Generic (PLEG): container finished" podID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerID="168baa1b15e59ec3108e5f48f196ea479f024aca734204d37970a7b81537d356" exitCode=60 Oct 31 13:15:24.295700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.288245 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerDied","Data":"168baa1b15e59ec3108e5f48f196ea479f024aca734204d37970a7b81537d356"} Oct 31 13:15:24.295700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.288741 5074 scope.go:122] "RemoveContainer" containerID="168baa1b15e59ec3108e5f48f196ea479f024aca734204d37970a7b81537d356" Oct 31 13:15:24.295700 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:24.290243 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t_odh-ai-gateway-infra(2c1acca3-b662-4b50-8de7-4f3dfa151f2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" Oct 31 13:15:24.295700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.294019 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerStarted","Data":"c68af68bb94afbd696203ca1ff1c8e75e3445f71b7a71caeedb16f65dda784b8"} Oct 31 13:15:24.302899 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.299680 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" event={"ID":"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80","Type":"ContainerStarted","Data":"2d6a2c757ad1f77127a79047ff5f34c5c7654a12b7c613de34dc12f514a6ee85"} Oct 31 13:15:24.307988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.306527 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" event={"ID":"721a5252-6907-4590-a6fc-bd7e1dc2a780","Type":"ContainerStarted","Data":"560480317db3cff6563eb65bf325b69d604392d7775666b4587acd7b0cf7fcb5"} Oct 31 13:15:24.307988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.307005 5074 scope.go:122] "RemoveContainer" containerID="560480317db3cff6563eb65bf325b69d604392d7775666b4587acd7b0cf7fcb5" Oct 31 13:15:24.307988 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:24.307421 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj_odh-ai-gateway-infra(721a5252-6907-4590-a6fc-bd7e1dc2a780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" Oct 31 13:15:24.315521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.310432 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" event={"ID":"28505a68-5e10-4461-a95e-5680564eb5b7","Type":"ContainerStarted","Data":"154c31fc7565d36de88206e5b8e73687c95d29e2300e260e3e45eba4d4589aef"} Oct 31 13:15:24.315521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.313866 5074 generic.go:356] "Generic (PLEG): container finished" podID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerID="5564161a91c1c508f024d994f323bfbf4eeabcf60170df8f0a9519f49c6cbffb" exitCode=60 Oct 31 13:15:24.315521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.313952 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" event={"ID":"f659ac7d-4b1e-429b-8f44-d41fab077a70","Type":"ContainerDied","Data":"5564161a91c1c508f024d994f323bfbf4eeabcf60170df8f0a9519f49c6cbffb"} Oct 31 13:15:24.315521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.314345 5074 scope.go:122] "RemoveContainer" containerID="5564161a91c1c508f024d994f323bfbf4eeabcf60170df8f0a9519f49c6cbffb" Oct 31 13:15:24.315521 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:24.314742 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp_odh-ai-gateway-infra(f659ac7d-4b1e-429b-8f44-d41fab077a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" Oct 31 13:15:24.321559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.320662 5074 generic.go:356] "Generic (PLEG): container finished" podID="dddae608-97bc-47b7-882d-710474720e95" containerID="cfe66fb6b043d985c446ad7386d05f5d2e3c74e7f91ec695c3a746e3cccf1791" exitCode=60 Oct 31 13:15:24.321559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.320789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerDied","Data":"cfe66fb6b043d985c446ad7386d05f5d2e3c74e7f91ec695c3a746e3cccf1791"} Oct 31 13:15:24.321559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.321217 5074 scope.go:122] "RemoveContainer" containerID="cfe66fb6b043d985c446ad7386d05f5d2e3c74e7f91ec695c3a746e3cccf1791" Oct 31 13:15:24.321836 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:24.321598 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m_odh-ai-gateway-infra(dddae608-97bc-47b7-882d-710474720e95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" podUID="dddae608-97bc-47b7-882d-710474720e95" Oct 31 13:15:24.325362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.325337 5074 generic.go:356] "Generic (PLEG): container finished" podID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerID="54ae67eff2732cc59e7ad9b3777ea92b63e7f55719ebe3c49976e6edb9254b1f" exitCode=60 Oct 31 13:15:24.325473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.325377 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerDied","Data":"54ae67eff2732cc59e7ad9b3777ea92b63e7f55719ebe3c49976e6edb9254b1f"} Oct 31 13:15:24.325843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.325823 5074 scope.go:122] "RemoveContainer" containerID="54ae67eff2732cc59e7ad9b3777ea92b63e7f55719ebe3c49976e6edb9254b1f" Oct 31 13:15:24.326237 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:24.326216 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8_odh-ai-gateway-infra(1abcd738-c940-44ba-9b90-6081d4a6b861)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" Oct 31 13:15:24.328082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.328035 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp"] Oct 31 13:15:24.374734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.374701 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t"] Oct 31 13:15:24.419347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.419309 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m"] Oct 31 13:15:24.486712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.486682 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8"] Oct 31 13:15:24.554918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.554879 5074 scope.go:122] "RemoveContainer" containerID="760a21577064783e288b8c07e765e513b3f9ac542aee5f4ff64b619623f0d2b8" Oct 31 13:15:24.575467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.575443 5074 scope.go:122] "RemoveContainer" containerID="4ca37820c350da71a859b56b7036bc55c6c80c1ec3f5cb2182a75f580d041f5a" Oct 31 13:15:24.593729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.593705 5074 scope.go:122] "RemoveContainer" containerID="3d051541a6be65dad13f8399a6b288ac6b9f28150764c0676c6ae2a7d00ef86c" Oct 31 13:15:24.689680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.688490 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:24.698893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.698865 5074 scope.go:122] "RemoveContainer" containerID="2b04f5c671b18b968c6fbb164932dc9c93167f3d1b610f83c21bfe5b7f4156c1" Oct 31 13:15:24.808649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.808505 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k"] Oct 31 13:15:24.814898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.814865 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0c1fbb-29890875-pcr4k"] Oct 31 13:15:24.846811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.846783 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s"] Oct 31 13:15:24.852133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.852109 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a434213-99ab-4ceb-8a6e-424927391e99-kube-api-access-g78kc\" (UniqueName: \"kubernetes.io/projected/6a434213-99ab-4ceb-8a6e-424927391e99-kube-api-access-g78kc\") pod \"6a434213-99ab-4ceb-8a6e-424927391e99\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " Oct 31 13:15:24.852268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.852173 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a434213-99ab-4ceb-8a6e-424927391e99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a434213-99ab-4ceb-8a6e-424927391e99-maas-api-service-ca\") pod \"6a434213-99ab-4ceb-8a6e-424927391e99\" (UID: \"6a434213-99ab-4ceb-8a6e-424927391e99\") " Oct 31 13:15:24.852756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.852712 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a434213-99ab-4ceb-8a6e-424927391e99-maas-api-service-ca" pod "6a434213-99ab-4ceb-8a6e-424927391e99" (UID: "6a434213-99ab-4ceb-8a6e-424927391e99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:24.853543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.853489 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a434213-99ab-4ceb-8a6e-424927391e99-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:24.854930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.854904 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a434213-99ab-4ceb-8a6e-424927391e99-kube-api-access-g78kc" pod "6a434213-99ab-4ceb-8a6e-424927391e99" (UID: "6a434213-99ab-4ceb-8a6e-424927391e99"). InnerVolumeSpecName "kube-api-access-g78kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:24.859276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.859251 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890875-7k72s"] Oct 31 13:15:24.931786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.931754 5074 scope.go:122] "RemoveContainer" containerID="52c83985aa622aa988f6930473c778db492592aa13ad432d0bf61841b6d613e7" Oct 31 13:15:24.955424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:24.955401 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g78kc\" (UniqueName: \"kubernetes.io/projected/6a434213-99ab-4ceb-8a6e-424927391e99-kube-api-access-g78kc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.221105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.221076 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272081 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-data\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272477 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-data" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272623 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-envoy\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272679 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-certs\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272737 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-istio-token\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-istio-token\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272846 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/316711a0-4531-404b-9e48-8c7824f93978-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316711a0-4531-404b-9e48-8c7824f93978-istio-podinfo\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272925 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-credential-socket\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.272982 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-socket\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.273025 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-kube-api-access-pldwq\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-kube-api-access-pldwq\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.273165 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316711a0-4531-404b-9e48-8c7824f93978-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316711a0-4531-404b-9e48-8c7824f93978-istiod-ca-cert\") pod \"316711a0-4531-404b-9e48-8c7824f93978\" (UID: \"316711a0-4531-404b-9e48-8c7824f93978\") " Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.273931 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.274302 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316711a0-4531-404b-9e48-8c7824f93978-istiod-ca-cert" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.274625 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-socket" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:25.275424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.274838 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-certs" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:25.276508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.275733 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-credential-socket" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:25.295143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.295102 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-envoy" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:15:25.295311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.295272 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/316711a0-4531-404b-9e48-8c7824f93978-istio-podinfo" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:15:25.295458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.295436 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-istio-token" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.295666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.295647 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-kube-api-access-pldwq" pod "316711a0-4531-404b-9e48-8c7824f93978" (UID: "316711a0-4531-404b-9e48-8c7824f93978"). InnerVolumeSpecName "kube-api-access-pldwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.343661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.343612 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" event={"ID":"d6f9576c-b31b-4cb0-84b9-32a6c471ab16","Type":"ContainerStarted","Data":"99408f56efeb865531b35a0953bc2b5ed8cfe15e71188fc2bcd3a7e6746ea0c8"} Oct 31 13:15:25.346976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.346952 5074 generic.go:356] "Generic (PLEG): container finished" podID="316711a0-4531-404b-9e48-8c7824f93978" containerID="c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47" exitCode=0 Oct 31 13:15:25.347077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.347020 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" Oct 31 13:15:25.347077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.347036 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" event={"ID":"316711a0-4531-404b-9e48-8c7824f93978","Type":"ContainerDied","Data":"c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47"} Oct 31 13:15:25.347234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.347097 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7" event={"ID":"316711a0-4531-404b-9e48-8c7824f93978","Type":"ContainerDied","Data":"9cb871cb133eece43dcad890967510fa9442516afa258a8d7d5e2a43d5291762"} Oct 31 13:15:25.347234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.347129 5074 scope.go:122] "RemoveContainer" containerID="c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47" Oct 31 13:15:25.355758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.355734 5074 generic.go:356] "Generic (PLEG): container finished" podID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerID="c68af68bb94afbd696203ca1ff1c8e75e3445f71b7a71caeedb16f65dda784b8" exitCode=60 Oct 31 13:15:25.355848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.355827 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerDied","Data":"c68af68bb94afbd696203ca1ff1c8e75e3445f71b7a71caeedb16f65dda784b8"} Oct 31 13:15:25.356247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.356227 5074 scope.go:122] "RemoveContainer" containerID="c68af68bb94afbd696203ca1ff1c8e75e3445f71b7a71caeedb16f65dda784b8" Oct 31 13:15:25.356668 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.356601 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk_odh-ai-gateway-infra(53dd0ac7-66bd-4841-aa0e-0a0b41294dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" Oct 31 13:15:25.358281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.358263 5074 generic.go:356] "Generic (PLEG): container finished" podID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerID="2d6a2c757ad1f77127a79047ff5f34c5c7654a12b7c613de34dc12f514a6ee85" exitCode=60 Oct 31 13:15:25.358444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.358420 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" event={"ID":"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80","Type":"ContainerDied","Data":"2d6a2c757ad1f77127a79047ff5f34c5c7654a12b7c613de34dc12f514a6ee85"} Oct 31 13:15:25.360452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.360358 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerStarted","Data":"b6549e28a0b17b36116747dd051d20e495008965c711e4474b3059d3f155df98"} Oct 31 13:15:25.360661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.360587 5074 scope.go:122] "RemoveContainer" containerID="2d6a2c757ad1f77127a79047ff5f34c5c7654a12b7c613de34dc12f514a6ee85" Oct 31 13:15:25.361782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.361759 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" event={"ID":"6a434213-99ab-4ceb-8a6e-424927391e99","Type":"ContainerDied","Data":"6372dad55f4a22e9ebae0f5227b15718ad6ee4f1c7831c2c42180d8af2867364"} Oct 31 13:15:25.361860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.361845 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46" Oct 31 13:15:25.362001 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.361973 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4_odh-ai-gateway-infra(e1ac994b-957d-47e5-8c0d-20dbc4c2ea80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" Oct 31 13:15:25.369775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.368768 5074 generic.go:356] "Generic (PLEG): container finished" podID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerID="560480317db3cff6563eb65bf325b69d604392d7775666b4587acd7b0cf7fcb5" exitCode=60 Oct 31 13:15:25.369775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.368867 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" event={"ID":"721a5252-6907-4590-a6fc-bd7e1dc2a780","Type":"ContainerDied","Data":"560480317db3cff6563eb65bf325b69d604392d7775666b4587acd7b0cf7fcb5"} Oct 31 13:15:25.369775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.369326 5074 scope.go:122] "RemoveContainer" containerID="560480317db3cff6563eb65bf325b69d604392d7775666b4587acd7b0cf7fcb5" Oct 31 13:15:25.369775 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.369715 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj_odh-ai-gateway-infra(721a5252-6907-4590-a6fc-bd7e1dc2a780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384342 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316711a0-4531-404b-9e48-8c7824f93978-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384373 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384389 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384403 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pldwq\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-kube-api-access-pldwq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384420 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316711a0-4531-404b-9e48-8c7824f93978-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384435 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384450 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316711a0-4531-404b-9e48-8c7824f93978-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.385373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.384465 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316711a0-4531-404b-9e48-8c7824f93978-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.388806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.388157 5074 generic.go:356] "Generic (PLEG): container finished" podID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerID="6d5935e3b8febfdd437d40da2370cfd9c514244f09fb153380e0986eecd06901" exitCode=60 Oct 31 13:15:25.388806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.388255 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" event={"ID":"095b3e51-cd04-44ef-9798-fe0e94c19bef","Type":"ContainerDied","Data":"6d5935e3b8febfdd437d40da2370cfd9c514244f09fb153380e0986eecd06901"} Oct 31 13:15:25.388806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.388650 5074 scope.go:122] "RemoveContainer" containerID="6d5935e3b8febfdd437d40da2370cfd9c514244f09fb153380e0986eecd06901" Oct 31 13:15:25.389047 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.388958 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm_odh-ai-gateway-infra(095b3e51-cd04-44ef-9798-fe0e94c19bef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.395854 5074 generic.go:356] "Generic (PLEG): container finished" podID="2af9234f-cf40-4883-8d91-c392be69d425" containerID="5f029d3792125db643e462b84f1613ea0f0c3121e4a545a67511d9f12f79a68d" exitCode=60 Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.395941 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" event={"ID":"2af9234f-cf40-4883-8d91-c392be69d425","Type":"ContainerDied","Data":"5f029d3792125db643e462b84f1613ea0f0c3121e4a545a67511d9f12f79a68d"} Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.396385 5074 scope.go:122] "RemoveContainer" containerID="5f029d3792125db643e462b84f1613ea0f0c3121e4a545a67511d9f12f79a68d" Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.396737 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4_odh-ai-gateway-infra(2af9234f-cf40-4883-8d91-c392be69d425)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" podUID="2af9234f-cf40-4883-8d91-c392be69d425" Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.399037 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4"] Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.400322 5074 generic.go:356] "Generic (PLEG): container finished" podID="28505a68-5e10-4461-a95e-5680564eb5b7" containerID="154c31fc7565d36de88206e5b8e73687c95d29e2300e260e3e45eba4d4589aef" exitCode=60 Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.400410 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" event={"ID":"28505a68-5e10-4461-a95e-5680564eb5b7","Type":"ContainerDied","Data":"154c31fc7565d36de88206e5b8e73687c95d29e2300e260e3e45eba4d4589aef"} Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.400885 5074 scope.go:122] "RemoveContainer" containerID="154c31fc7565d36de88206e5b8e73687c95d29e2300e260e3e45eba4d4589aef" Oct 31 13:15:25.403065 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.401213 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm_odh-ai-gateway-infra(28505a68-5e10-4461-a95e-5680564eb5b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" Oct 31 13:15:25.443188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.436692 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj"] Oct 31 13:15:25.508270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.508216 5074 scope.go:122] "RemoveContainer" containerID="e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269" Oct 31 13:15:25.530529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.529711 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm"] Oct 31 13:15:25.536352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.536333 5074 scope.go:122] "RemoveContainer" containerID="c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47" Oct 31 13:15:25.536740 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.536715 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47\": container with ID starting with c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47 not found: ID does not exist" containerID="c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47" Oct 31 13:15:25.536815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.536756 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47"} err="failed to get container status \"c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47\": rpc error: code = NotFound desc = could not find container \"c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47\": container with ID starting with c367c1cdeddbee41dc0220b6b3c969a1a052036ce5addb123ad3251548c01e47 not found: ID does not exist" Oct 31 13:15:25.536815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.536780 5074 scope.go:122] "RemoveContainer" containerID="e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269" Oct 31 13:15:25.537243 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.537145 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269\": container with ID starting with e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269 not found: ID does not exist" containerID="e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269" Oct 31 13:15:25.537243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.537171 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269"} err="failed to get container status \"e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269\": rpc error: code = NotFound desc = could not find container \"e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269\": container with ID starting with e8cdeaa1a757726509d40a6277b0ce14b0d7e56aaef33979505f485398c2d269 not found: ID does not exist" Oct 31 13:15:25.537243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.537185 5074 scope.go:122] "RemoveContainer" containerID="31b58a3cf85820d74c87e0274f0ff905992969142f69c576cc8d398f65d7eb62" Oct 31 13:15:25.588590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.588493 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:25.604386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.604299 5074 scope.go:122] "RemoveContainer" containerID="7427610d0f14747d64ac0525ecb78ab5362423215c0dc5cb3c8f7ecea701d5ff" Oct 31 13:15:25.638484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.638202 5074 scope.go:122] "RemoveContainer" containerID="234b0ac0c0fdc8b7678fbac4c52c19616daef8ccf55dead705a0bae6943eb5c1" Oct 31 13:15:25.653550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.653527 5074 scope.go:122] "RemoveContainer" containerID="04bdb38c7fda94f4ece7bf3829ea890ae81ee43473ae98064e4781d114f23d0a" Oct 31 13:15:25.682210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.682188 5074 scope.go:122] "RemoveContainer" containerID="08c294588b733e5f29a489702da27b1fe9edb2c4cd538bab71efdd4c12b0815b" Oct 31 13:15:25.684162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.683355 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:25.689163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.689102 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abcd738-c940-44ba-9b90-6081d4a6b861-kube-api-access-mxqc5\" (UniqueName: \"kubernetes.io/projected/1abcd738-c940-44ba-9b90-6081d4a6b861-kube-api-access-mxqc5\") pod \"1abcd738-c940-44ba-9b90-6081d4a6b861\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " Oct 31 13:15:25.689267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.689252 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abcd738-c940-44ba-9b90-6081d4a6b861-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abcd738-c940-44ba-9b90-6081d4a6b861-maas-api-service-ca\") pod \"1abcd738-c940-44ba-9b90-6081d4a6b861\" (UID: \"1abcd738-c940-44ba-9b90-6081d4a6b861\") " Oct 31 13:15:25.689856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.689835 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abcd738-c940-44ba-9b90-6081d4a6b861-maas-api-service-ca" pod "1abcd738-c940-44ba-9b90-6081d4a6b861" (UID: "1abcd738-c940-44ba-9b90-6081d4a6b861"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:25.691456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.690988 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abcd738-c940-44ba-9b90-6081d4a6b861-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.693204 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.693177 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abcd738-c940-44ba-9b90-6081d4a6b861-kube-api-access-mxqc5" pod "1abcd738-c940-44ba-9b90-6081d4a6b861" (UID: "1abcd738-c940-44ba-9b90-6081d4a6b861"). InnerVolumeSpecName "kube-api-access-mxqc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.728659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.728592 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk"] Oct 31 13:15:25.739393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.739356 5074 scope.go:122] "RemoveContainer" containerID="c16ee75180b71119049818322127ee5b0fafc095a70c96dbeb0a52dea8809158" Oct 31 13:15:25.749574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.749517 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46"] Oct 31 13:15:25.766138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.766110 5074 scope.go:122] "RemoveContainer" containerID="1b67ccd202545989d9ffadd91b6738958a80e99ff6fe48cb84d3211a94be1e77" Oct 31 13:15:25.767366 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.767148 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:25.773328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.771432 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890875-l5k46"] Oct 31 13:15:25.779885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.779810 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:25.794665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.792981 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f659ac7d-4b1e-429b-8f44-d41fab077a70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f659ac7d-4b1e-429b-8f44-d41fab077a70-maas-api-service-ca\") pod \"f659ac7d-4b1e-429b-8f44-d41fab077a70\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " Oct 31 13:15:25.794665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.793137 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f659ac7d-4b1e-429b-8f44-d41fab077a70-kube-api-access-pshdz\" (UniqueName: \"kubernetes.io/projected/f659ac7d-4b1e-429b-8f44-d41fab077a70-kube-api-access-pshdz\") pod \"f659ac7d-4b1e-429b-8f44-d41fab077a70\" (UID: \"f659ac7d-4b1e-429b-8f44-d41fab077a70\") " Oct 31 13:15:25.794665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.793241 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-kube-api-access-5lqc7\" (UniqueName: \"kubernetes.io/projected/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-kube-api-access-5lqc7\") pod \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " Oct 31 13:15:25.794665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.793406 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-maas-api-service-ca\") pod \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\" (UID: \"2c1acca3-b662-4b50-8de7-4f3dfa151f2f\") " Oct 31 13:15:25.794665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.794047 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxqc5\" (UniqueName: \"kubernetes.io/projected/1abcd738-c940-44ba-9b90-6081d4a6b861-kube-api-access-mxqc5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.795137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.795113 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f659ac7d-4b1e-429b-8f44-d41fab077a70-maas-api-service-ca" pod "f659ac7d-4b1e-429b-8f44-d41fab077a70" (UID: "f659ac7d-4b1e-429b-8f44-d41fab077a70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:25.795416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.795398 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-maas-api-service-ca" pod "2c1acca3-b662-4b50-8de7-4f3dfa151f2f" (UID: "2c1acca3-b662-4b50-8de7-4f3dfa151f2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:25.805906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.805881 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-kube-api-access-5lqc7" pod "2c1acca3-b662-4b50-8de7-4f3dfa151f2f" (UID: "2c1acca3-b662-4b50-8de7-4f3dfa151f2f"). InnerVolumeSpecName "kube-api-access-5lqc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.814834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.814801 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f659ac7d-4b1e-429b-8f44-d41fab077a70-kube-api-access-pshdz" pod "f659ac7d-4b1e-429b-8f44-d41fab077a70" (UID: "f659ac7d-4b1e-429b-8f44-d41fab077a70"). InnerVolumeSpecName "kube-api-access-pshdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.856299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.856262 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7"] Oct 31 13:15:25.860103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.860064 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-fa8c3f-openshift-default-7d794b76cc-4cbf7"] Oct 31 13:15:25.894570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.894530 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddae608-97bc-47b7-882d-710474720e95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddae608-97bc-47b7-882d-710474720e95-maas-api-service-ca\") pod \"dddae608-97bc-47b7-882d-710474720e95\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " Oct 31 13:15:25.894879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.894852 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddae608-97bc-47b7-882d-710474720e95-kube-api-access-vsmpk\" (UniqueName: \"kubernetes.io/projected/dddae608-97bc-47b7-882d-710474720e95-kube-api-access-vsmpk\") pod \"dddae608-97bc-47b7-882d-710474720e95\" (UID: \"dddae608-97bc-47b7-882d-710474720e95\") " Oct 31 13:15:25.895122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.895091 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddae608-97bc-47b7-882d-710474720e95-maas-api-service-ca" pod "dddae608-97bc-47b7-882d-710474720e95" (UID: "dddae608-97bc-47b7-882d-710474720e95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:25.895781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.895756 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pshdz\" (UniqueName: \"kubernetes.io/projected/f659ac7d-4b1e-429b-8f44-d41fab077a70-kube-api-access-pshdz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.895877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.895788 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddae608-97bc-47b7-882d-710474720e95-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.895877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.895804 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lqc7\" (UniqueName: \"kubernetes.io/projected/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-kube-api-access-5lqc7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.895877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.895818 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1acca3-b662-4b50-8de7-4f3dfa151f2f-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.895877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.895833 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f659ac7d-4b1e-429b-8f44-d41fab077a70-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:25.898865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.898832 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddae608-97bc-47b7-882d-710474720e95-kube-api-access-vsmpk" pod "dddae608-97bc-47b7-882d-710474720e95" (UID: "dddae608-97bc-47b7-882d-710474720e95"). InnerVolumeSpecName "kube-api-access-vsmpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:25.943272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.943180 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" path="/var/lib/kubelet/pods/1aafcb56-25d6-4d5e-83f0-98bb476db5a4/volumes" Oct 31 13:15:25.943889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.943865 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316711a0-4531-404b-9e48-8c7824f93978" path="/var/lib/kubelet/pods/316711a0-4531-404b-9e48-8c7824f93978/volumes" Oct 31 13:15:25.944365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.944343 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" path="/var/lib/kubelet/pods/6a434213-99ab-4ceb-8a6e-424927391e99/volumes" Oct 31 13:15:25.945091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.945050 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" path="/var/lib/kubelet/pods/af8fd503-9169-4d5a-aab0-799659a0be24/volumes" Oct 31 13:15:25.993247 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:25.993207 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:15:25.997671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:25.997646 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmpk\" (UniqueName: \"kubernetes.io/projected/dddae608-97bc-47b7-882d-710474720e95-kube-api-access-vsmpk\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:26.462656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.461751 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" event={"ID":"2c1acca3-b662-4b50-8de7-4f3dfa151f2f","Type":"ContainerDied","Data":"3a7a9142cff28d8f91f63d19e1ceb63fce6d70bee0ba74d2cab9bec917812d5f"} Oct 31 13:15:26.462656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.461830 5074 scope.go:122] "RemoveContainer" containerID="168baa1b15e59ec3108e5f48f196ea479f024aca734204d37970a7b81537d356" Oct 31 13:15:26.462656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.461956 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t" Oct 31 13:15:26.522001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.517411 5074 generic.go:356] "Generic (PLEG): container finished" podID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerID="b6549e28a0b17b36116747dd051d20e495008965c711e4474b3059d3f155df98" exitCode=60 Oct 31 13:15:26.522001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.517532 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerDied","Data":"b6549e28a0b17b36116747dd051d20e495008965c711e4474b3059d3f155df98"} Oct 31 13:15:26.522001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.518133 5074 scope.go:122] "RemoveContainer" containerID="b6549e28a0b17b36116747dd051d20e495008965c711e4474b3059d3f155df98" Oct 31 13:15:26.522001 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:26.518601 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96_odh-ai-gateway-infra(56c5cc9f-9c47-4155-855f-e8b9ad1a876e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" Oct 31 13:15:26.548387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.548353 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" event={"ID":"f659ac7d-4b1e-429b-8f44-d41fab077a70","Type":"ContainerDied","Data":"fb4ff96dfebff5ad647d63cc487f743e39b5ef20592f34ff00dc20197b0b59e2"} Oct 31 13:15:26.548623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.548606 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp" Oct 31 13:15:26.563005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.562974 5074 generic.go:356] "Generic (PLEG): container finished" podID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerID="9ebdd6273b2abf2cd82e1fd5a3fe65110cfd1adad83eb2cf0a1ad3633c61753d" exitCode=60 Oct 31 13:15:26.563853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.563825 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" event={"ID":"03a45b88-fafa-420f-a7ff-99abed7bd013","Type":"ContainerDied","Data":"9ebdd6273b2abf2cd82e1fd5a3fe65110cfd1adad83eb2cf0a1ad3633c61753d"} Oct 31 13:15:26.564543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.564524 5074 scope.go:122] "RemoveContainer" containerID="9ebdd6273b2abf2cd82e1fd5a3fe65110cfd1adad83eb2cf0a1ad3633c61753d" Oct 31 13:15:26.565160 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:26.565138 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d_odh-ai-gateway-infra(03a45b88-fafa-420f-a7ff-99abed7bd013)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" Oct 31 13:15:26.567643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.567598 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" event={"ID":"dddae608-97bc-47b7-882d-710474720e95","Type":"ContainerDied","Data":"6156ba012bfb7333255ffedd7ab1273368f85e4079d5388e3fc8036a01044102"} Oct 31 13:15:26.567892 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.567877 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m" Oct 31 13:15:26.591867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.587730 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96"] Oct 31 13:15:26.596655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.593707 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt"] Oct 31 13:15:26.608890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.608853 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" event={"ID":"1abcd738-c940-44ba-9b90-6081d4a6b861","Type":"ContainerDied","Data":"fa69467fb4526ba62a439fc6d9107f70f9673bf3781177f23c912cb286597f85"} Oct 31 13:15:26.614194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.609030 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8" Oct 31 13:15:26.614194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.611525 5074 generic.go:356] "Generic (PLEG): container finished" podID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerID="99408f56efeb865531b35a0953bc2b5ed8cfe15e71188fc2bcd3a7e6746ea0c8" exitCode=60 Oct 31 13:15:26.615983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.615960 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" event={"ID":"d6f9576c-b31b-4cb0-84b9-32a6c471ab16","Type":"ContainerDied","Data":"99408f56efeb865531b35a0953bc2b5ed8cfe15e71188fc2bcd3a7e6746ea0c8"} Oct 31 13:15:26.616559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.616539 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t"] Oct 31 13:15:26.637471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.625168 5074 scope.go:122] "RemoveContainer" containerID="e9abfacc2c7078177a9ac14a549f52b28b42f014f3b8efaf1abccbd883db30e5" Oct 31 13:15:26.639312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.639252 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890875-zlm4t"] Oct 31 13:15:26.647555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.647259 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4"] Oct 31 13:15:26.687225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.687097 5074 scope.go:122] "RemoveContainer" containerID="5564161a91c1c508f024d994f323bfbf4eeabcf60170df8f0a9519f49c6cbffb" Oct 31 13:15:26.695163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.693751 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm"] Oct 31 13:15:26.743409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.743207 5074 scope.go:122] "RemoveContainer" containerID="52c83985aa622aa988f6930473c778db492592aa13ad432d0bf61841b6d613e7" Oct 31 13:15:26.780893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.777112 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8"] Oct 31 13:15:26.791360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.791294 5074 scope.go:122] "RemoveContainer" containerID="cfe66fb6b043d985c446ad7386d05f5d2e3c74e7f91ec695c3a746e3cccf1791" Oct 31 13:15:26.793071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.793042 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c24e1-29890875-l7dl8"] Oct 31 13:15:26.846959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.815627 5074 scope.go:122] "RemoveContainer" containerID="54ae67eff2732cc59e7ad9b3777ea92b63e7f55719ebe3c49976e6edb9254b1f" Oct 31 13:15:26.898081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:26.897940 5074 scope.go:122] "RemoveContainer" containerID="4145d446eb452dac06ed3639ab5c2bf1981ac386e087db1e57a14ecc5049634c" Oct 31 13:15:27.000831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.000805 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp"] Oct 31 13:15:27.017490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.017462 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:27.020077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.020055 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:27.026464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.026333 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5dc570-29890875-4qnkp"] Oct 31 13:15:27.058238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.055997 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m"] Oct 31 13:15:27.058238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.056492 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:27.064305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.063228 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:27.064418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.064390 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6cef79-29890875-df44m"] Oct 31 13:15:27.071842 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.071814 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:27.101948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.101912 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-maas-api-service-ca\") pod \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " Oct 31 13:15:27.102124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.101965 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-kube-api-access-px9x8\" (UniqueName: \"kubernetes.io/projected/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-kube-api-access-px9x8\") pod \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\" (UID: \"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80\") " Oct 31 13:15:27.102304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.102279 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28505a68-5e10-4461-a95e-5680564eb5b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28505a68-5e10-4461-a95e-5680564eb5b7-maas-api-service-ca\") pod \"28505a68-5e10-4461-a95e-5680564eb5b7\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " Oct 31 13:15:27.102425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.102408 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28505a68-5e10-4461-a95e-5680564eb5b7-kube-api-access-n66b5\" (UniqueName: \"kubernetes.io/projected/28505a68-5e10-4461-a95e-5680564eb5b7-kube-api-access-n66b5\") pod \"28505a68-5e10-4461-a95e-5680564eb5b7\" (UID: \"28505a68-5e10-4461-a95e-5680564eb5b7\") " Oct 31 13:15:27.106083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.106041 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28505a68-5e10-4461-a95e-5680564eb5b7-maas-api-service-ca" pod "28505a68-5e10-4461-a95e-5680564eb5b7" (UID: "28505a68-5e10-4461-a95e-5680564eb5b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:27.106334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.106312 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-maas-api-service-ca" pod "e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" (UID: "e1ac994b-957d-47e5-8c0d-20dbc4c2ea80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:27.110763 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.110722 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-kube-api-access-px9x8" pod "e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" (UID: "e1ac994b-957d-47e5-8c0d-20dbc4c2ea80"). InnerVolumeSpecName "kube-api-access-px9x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.117980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.117939 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28505a68-5e10-4461-a95e-5680564eb5b7-kube-api-access-n66b5" pod "28505a68-5e10-4461-a95e-5680564eb5b7" (UID: "28505a68-5e10-4461-a95e-5680564eb5b7"). InnerVolumeSpecName "kube-api-access-n66b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.204065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.204038 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-maas-api-service-ca\") pod \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " Oct 31 13:15:27.204600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.204583 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-kube-api-access-cn9v8\" (UniqueName: \"kubernetes.io/projected/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-kube-api-access-cn9v8\") pod \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\" (UID: \"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3\") " Oct 31 13:15:27.204800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.204789 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721a5252-6907-4590-a6fc-bd7e1dc2a780-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721a5252-6907-4590-a6fc-bd7e1dc2a780-maas-api-service-ca\") pod \"721a5252-6907-4590-a6fc-bd7e1dc2a780\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " Oct 31 13:15:27.204954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.204939 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af9234f-cf40-4883-8d91-c392be69d425-kube-api-access-bnxb7\" (UniqueName: \"kubernetes.io/projected/2af9234f-cf40-4883-8d91-c392be69d425-kube-api-access-bnxb7\") pod \"2af9234f-cf40-4883-8d91-c392be69d425\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " Oct 31 13:15:27.205116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.205081 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-maas-api-service-ca" pod "53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" (UID: "53dd0ac7-66bd-4841-aa0e-0a0b41294dd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:27.205178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.205091 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721a5252-6907-4590-a6fc-bd7e1dc2a780-kube-api-access-cx7n7\" (UniqueName: \"kubernetes.io/projected/721a5252-6907-4590-a6fc-bd7e1dc2a780-kube-api-access-cx7n7\") pod \"721a5252-6907-4590-a6fc-bd7e1dc2a780\" (UID: \"721a5252-6907-4590-a6fc-bd7e1dc2a780\") " Oct 31 13:15:27.205178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.205160 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721a5252-6907-4590-a6fc-bd7e1dc2a780-maas-api-service-ca" pod "721a5252-6907-4590-a6fc-bd7e1dc2a780" (UID: "721a5252-6907-4590-a6fc-bd7e1dc2a780"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:27.205284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.205205 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af9234f-cf40-4883-8d91-c392be69d425-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af9234f-cf40-4883-8d91-c392be69d425-maas-api-service-ca\") pod \"2af9234f-cf40-4883-8d91-c392be69d425\" (UID: \"2af9234f-cf40-4883-8d91-c392be69d425\") " Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.205922 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af9234f-cf40-4883-8d91-c392be69d425-maas-api-service-ca" pod "2af9234f-cf40-4883-8d91-c392be69d425" (UID: "2af9234f-cf40-4883-8d91-c392be69d425"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.206095 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721a5252-6907-4590-a6fc-bd7e1dc2a780-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.206111 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.206121 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9x8\" (UniqueName: \"kubernetes.io/projected/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80-kube-api-access-px9x8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.206130 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af9234f-cf40-4883-8d91-c392be69d425-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.206138 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.206147 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28505a68-5e10-4461-a95e-5680564eb5b7-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.206158 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n66b5\" (UniqueName: \"kubernetes.io/projected/28505a68-5e10-4461-a95e-5680564eb5b7-kube-api-access-n66b5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.212680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.211105 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-kube-api-access-cn9v8" pod "53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" (UID: "53dd0ac7-66bd-4841-aa0e-0a0b41294dd3"). InnerVolumeSpecName "kube-api-access-cn9v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.213321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.213224 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:27.218680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.218599 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af9234f-cf40-4883-8d91-c392be69d425-kube-api-access-bnxb7" pod "2af9234f-cf40-4883-8d91-c392be69d425" (UID: "2af9234f-cf40-4883-8d91-c392be69d425"). InnerVolumeSpecName "kube-api-access-bnxb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.222743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.222704 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721a5252-6907-4590-a6fc-bd7e1dc2a780-kube-api-access-cx7n7" pod "721a5252-6907-4590-a6fc-bd7e1dc2a780" (UID: "721a5252-6907-4590-a6fc-bd7e1dc2a780"). InnerVolumeSpecName "kube-api-access-cx7n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.231736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.230197 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:27.307265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.307224 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-kube-api-access-t56sr\" (UniqueName: \"kubernetes.io/projected/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-kube-api-access-t56sr\") pod \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " Oct 31 13:15:27.307458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.307440 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-maas-api-service-ca\") pod \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\" (UID: \"d6f9576c-b31b-4cb0-84b9-32a6c471ab16\") " Oct 31 13:15:27.308194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.308158 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-maas-api-service-ca" pod "d6f9576c-b31b-4cb0-84b9-32a6c471ab16" (UID: "d6f9576c-b31b-4cb0-84b9-32a6c471ab16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:27.308859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.308795 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095b3e51-cd04-44ef-9798-fe0e94c19bef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b3e51-cd04-44ef-9798-fe0e94c19bef-maas-api-service-ca\") pod \"095b3e51-cd04-44ef-9798-fe0e94c19bef\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " Oct 31 13:15:27.308959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.308894 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095b3e51-cd04-44ef-9798-fe0e94c19bef-kube-api-access-g6pl2\" (UniqueName: \"kubernetes.io/projected/095b3e51-cd04-44ef-9798-fe0e94c19bef-kube-api-access-g6pl2\") pod \"095b3e51-cd04-44ef-9798-fe0e94c19bef\" (UID: \"095b3e51-cd04-44ef-9798-fe0e94c19bef\") " Oct 31 13:15:27.309326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.309295 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095b3e51-cd04-44ef-9798-fe0e94c19bef-maas-api-service-ca" pod "095b3e51-cd04-44ef-9798-fe0e94c19bef" (UID: "095b3e51-cd04-44ef-9798-fe0e94c19bef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:27.309798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.309771 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b3e51-cd04-44ef-9798-fe0e94c19bef-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.309864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.309803 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn9v8\" (UniqueName: \"kubernetes.io/projected/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3-kube-api-access-cn9v8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.309864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.309819 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxb7\" (UniqueName: \"kubernetes.io/projected/2af9234f-cf40-4883-8d91-c392be69d425-kube-api-access-bnxb7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.309864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.309833 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7n7\" (UniqueName: \"kubernetes.io/projected/721a5252-6907-4590-a6fc-bd7e1dc2a780-kube-api-access-cx7n7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.309864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.309847 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.314138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.314109 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-kube-api-access-t56sr" pod "d6f9576c-b31b-4cb0-84b9-32a6c471ab16" (UID: "d6f9576c-b31b-4cb0-84b9-32a6c471ab16"). InnerVolumeSpecName "kube-api-access-t56sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.316178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.316149 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095b3e51-cd04-44ef-9798-fe0e94c19bef-kube-api-access-g6pl2" pod "095b3e51-cd04-44ef-9798-fe0e94c19bef" (UID: "095b3e51-cd04-44ef-9798-fe0e94c19bef"). InnerVolumeSpecName "kube-api-access-g6pl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:27.411835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.411797 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6pl2\" (UniqueName: \"kubernetes.io/projected/095b3e51-cd04-44ef-9798-fe0e94c19bef-kube-api-access-g6pl2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.411835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.411836 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56sr\" (UniqueName: \"kubernetes.io/projected/d6f9576c-b31b-4cb0-84b9-32a6c471ab16-kube-api-access-t56sr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:27.630431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.630347 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" Oct 31 13:15:27.630431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.630367 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk" event={"ID":"53dd0ac7-66bd-4841-aa0e-0a0b41294dd3","Type":"ContainerDied","Data":"a48019a62d294ae63926b14caa5203ddb0e8201acecbd6a99dfb321ad2b84f4f"} Oct 31 13:15:27.630968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.630545 5074 scope.go:122] "RemoveContainer" containerID="c68af68bb94afbd696203ca1ff1c8e75e3445f71b7a71caeedb16f65dda784b8" Oct 31 13:15:27.632232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.632204 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" event={"ID":"e1ac994b-957d-47e5-8c0d-20dbc4c2ea80","Type":"ContainerDied","Data":"db2dfcf7097558c4651a16d1235a7b16b337b9713d0eaa69ceb234b0c732d9fa"} Oct 31 13:15:27.632364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.632217 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4" Oct 31 13:15:27.640279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.640259 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" Oct 31 13:15:27.640400 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.640289 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj" event={"ID":"721a5252-6907-4590-a6fc-bd7e1dc2a780","Type":"ContainerDied","Data":"7964efad0ea6f9764cf729bc72b36518af9ed54359630c4c1c1908a834f7889b"} Oct 31 13:15:27.642127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.642100 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" event={"ID":"095b3e51-cd04-44ef-9798-fe0e94c19bef","Type":"ContainerDied","Data":"0ef61e3a3775f05b2730eddd39daa4b15c6b55ef59a4ac6f4ef8f338d6e42143"} Oct 31 13:15:27.642270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.642195 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm" Oct 31 13:15:27.643955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.643934 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" Oct 31 13:15:27.644922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.644103 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4" event={"ID":"2af9234f-cf40-4883-8d91-c392be69d425","Type":"ContainerDied","Data":"6f4533b7bb9a97931d6e1a4c47a849823c8189d80df2eced51eacbb5ed51f81b"} Oct 31 13:15:27.648961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.647103 5074 scope.go:122] "RemoveContainer" containerID="2d6a2c757ad1f77127a79047ff5f34c5c7654a12b7c613de34dc12f514a6ee85" Oct 31 13:15:27.648961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.648300 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" Oct 31 13:15:27.648961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.648315 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm" event={"ID":"28505a68-5e10-4461-a95e-5680564eb5b7","Type":"ContainerDied","Data":"b9b92d604fcfdd50c28869c966f675f52d356751f23d4d3d281de93bd3ec8b7d"} Oct 31 13:15:27.670004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.669873 5074 scope.go:122] "RemoveContainer" containerID="560480317db3cff6563eb65bf325b69d604392d7775666b4587acd7b0cf7fcb5" Oct 31 13:15:27.671395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.671370 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" event={"ID":"d6f9576c-b31b-4cb0-84b9-32a6c471ab16","Type":"ContainerDied","Data":"0cc84fea818346f6c71d99842e9ebde1cfd152e41f38b4bd6d25e623145384fc"} Oct 31 13:15:27.671501 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.671412 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt" Oct 31 13:15:27.685563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.685536 5074 scope.go:122] "RemoveContainer" containerID="6d5935e3b8febfdd437d40da2370cfd9c514244f09fb153380e0986eecd06901" Oct 31 13:15:27.698839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.698816 5074 scope.go:122] "RemoveContainer" containerID="5f029d3792125db643e462b84f1613ea0f0c3121e4a545a67511d9f12f79a68d" Oct 31 13:15:27.710883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.710824 5074 scope.go:122] "RemoveContainer" containerID="154c31fc7565d36de88206e5b8e73687c95d29e2300e260e3e45eba4d4589aef" Oct 31 13:15:27.731924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.731885 5074 scope.go:122] "RemoveContainer" containerID="99408f56efeb865531b35a0953bc2b5ed8cfe15e71188fc2bcd3a7e6746ea0c8" Oct 31 13:15:27.755598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.755567 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8"] Oct 31 13:15:27.757193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757168 5074 cpu_manager.go:414] "containerMap: removing container" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.757193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757193 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757211 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757218 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757228 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757236 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757250 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757258 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757265 5074 cpu_manager.go:414] "containerMap: removing container" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757272 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757283 5074 cpu_manager.go:414] "containerMap: removing container" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757289 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757300 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757307 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757325 5074 cpu_manager.go:414] "containerMap: removing container" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757332 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757345 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757352 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.757369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757376 5074 cpu_manager.go:414] "containerMap: removing container" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757384 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757396 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757403 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757413 5074 cpu_manager.go:414] "containerMap: removing container" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757420 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757433 5074 cpu_manager.go:414] "containerMap: removing container" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757440 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757453 5074 cpu_manager.go:414] "containerMap: removing container" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757460 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757471 5074 cpu_manager.go:414] "containerMap: removing container" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757478 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757496 5074 cpu_manager.go:414] "containerMap: removing container" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757525 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757542 5074 cpu_manager.go:414] "containerMap: removing container" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757548 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757560 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757567 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757578 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757586 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757599 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757605 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757613 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757621 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757655 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757663 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757673 5074 cpu_manager.go:414] "containerMap: removing container" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757680 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757696 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757703 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757714 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d7399962-c419-4d22-85f0-6364c37e8379" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757721 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7399962-c419-4d22-85f0-6364c37e8379" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757736 5074 cpu_manager.go:414] "containerMap: removing container" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757744 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757753 5074 cpu_manager.go:414] "containerMap: removing container" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757759 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757774 5074 cpu_manager.go:414] "containerMap: removing container" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757780 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757794 5074 cpu_manager.go:414] "containerMap: removing container" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757802 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757815 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757822 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757832 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d7399962-c419-4d22-85f0-6364c37e8379" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757839 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7399962-c419-4d22-85f0-6364c37e8379" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757847 5074 cpu_manager.go:414] "containerMap: removing container" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757854 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757869 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757876 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757889 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757897 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757925 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.757933 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758186 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758210 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758223 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758234 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758247 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758255 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758265 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758276 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758283 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758292 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758301 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758310 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758318 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758328 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758335 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758343 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758352 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758364 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758373 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758381 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758391 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.758347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758403 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758413 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758421 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758431 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758441 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758451 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758462 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758470 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7399962-c419-4d22-85f0-6364c37e8379" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758477 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758487 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758496 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758507 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758519 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758527 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7399962-c419-4d22-85f0-6364c37e8379" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758536 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758546 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758555 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758564 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.761198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.758572 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.775210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.774480 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.789145 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9"] Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794102 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794125 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794142 5074 cpu_manager.go:414] "containerMap: removing container" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794150 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddae608-97bc-47b7-882d-710474720e95" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794172 5074 cpu_manager.go:414] "containerMap: removing container" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794180 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794219 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794225 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794236 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794243 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794257 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794264 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a434213-99ab-4ceb-8a6e-424927391e99" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794282 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794288 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794304 5074 cpu_manager.go:414] "containerMap: removing container" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794310 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8fd503-9169-4d5a-aab0-799659a0be24" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794329 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794335 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794394 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794401 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af9234f-cf40-4883-8d91-c392be69d425" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794417 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794424 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794671 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="316711a0-4531-404b-9e48-8c7824f93978" containerName="istio-proxy" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794693 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794702 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794716 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794732 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aafcb56-25d6-4d5e-83f0-98bb476db5a4" containerName="cleanup" Oct 31 13:15:27.795551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.794745 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" containerName="cleanup" Oct 31 13:15:27.810950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.799117 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2ef686\"" Oct 31 13:15:27.811326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.811278 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2ef686-dockercfg-749rd\"" Oct 31 13:15:27.816247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.815275 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8"] Oct 31 13:15:27.816247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.815318 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:15:27.816247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.815348 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9"] Oct 31 13:15:27.816247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.815372 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt"] Oct 31 13:15:27.816247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:27.815460 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.070990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.070819 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf2qs\" (UniqueName: \"kubernetes.io/projected/c8d9c999-6423-4ed8-9b67-47eda9032560-kube-api-access-hf2qs\") pod \"payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:28.094084 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.093949 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqlb\" (UniqueName: \"kubernetes.io/projected/4f7723da-6f6e-4f6b-ada0-76fb6af99085-kube-api-access-rfqlb\") pod \"payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.095479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.094964 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7723da-6f6e-4f6b-ada0-76fb6af99085-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.095910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.095891 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d9c999-6423-4ed8-9b67-47eda9032560-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:28.161362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.161320 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abcd738-c940-44ba-9b90-6081d4a6b861" path="/var/lib/kubelet/pods/1abcd738-c940-44ba-9b90-6081d4a6b861/volumes" Oct 31 13:15:28.179226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.178179 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:28.184099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.184068 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1acca3-b662-4b50-8de7-4f3dfa151f2f" path="/var/lib/kubelet/pods/2c1acca3-b662-4b50-8de7-4f3dfa151f2f/volumes" Oct 31 13:15:28.185461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.185432 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddae608-97bc-47b7-882d-710474720e95" path="/var/lib/kubelet/pods/dddae608-97bc-47b7-882d-710474720e95/volumes" Oct 31 13:15:28.186573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.186544 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f659ac7d-4b1e-429b-8f44-d41fab077a70" path="/var/lib/kubelet/pods/f659ac7d-4b1e-429b-8f44-d41fab077a70/volumes" Oct 31 13:15:28.187461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.187427 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ae9e7-29890875-86bjt"] Oct 31 13:15:28.187620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.187608 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4"] Oct 31 13:15:28.187744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.187723 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-377f60-29890875-wr2p4"] Oct 31 13:15:28.187848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.187838 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d"] Oct 31 13:15:28.189108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.189082 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj"] Oct 31 13:15:28.193976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.193942 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890875-2krfj"] Oct 31 13:15:28.197472 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.197441 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7723da-6f6e-4f6b-ada0-76fb6af99085-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.197582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.197503 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d9c999-6423-4ed8-9b67-47eda9032560-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:28.197731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.197710 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf2qs\" (UniqueName: \"kubernetes.io/projected/c8d9c999-6423-4ed8-9b67-47eda9032560-kube-api-access-hf2qs\") pod \"payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:28.197979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.197961 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqlb\" (UniqueName: \"kubernetes.io/projected/4f7723da-6f6e-4f6b-ada0-76fb6af99085-kube-api-access-rfqlb\") pod \"payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.199178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.198393 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7723da-6f6e-4f6b-ada0-76fb6af99085-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.199178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.198903 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d9c999-6423-4ed8-9b67-47eda9032560-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:28.200081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.200058 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm"] Oct 31 13:15:28.204235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.204217 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890875-xsdhm"] Oct 31 13:15:28.230712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.230676 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4"] Oct 31 13:15:28.233668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.233611 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf2qs\" (UniqueName: \"kubernetes.io/projected/c8d9c999-6423-4ed8-9b67-47eda9032560-kube-api-access-hf2qs\") pod \"payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:28.235989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.235966 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890875-fcxc4"] Oct 31 13:15:28.236626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.236607 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqlb\" (UniqueName: \"kubernetes.io/projected/4f7723da-6f6e-4f6b-ada0-76fb6af99085-kube-api-access-rfqlb\") pod \"payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.260532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.260506 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm"] Oct 31 13:15:28.269719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.267777 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-45bcad-29890875-qvtgm"] Oct 31 13:15:28.287686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.287608 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk"] Oct 31 13:15:28.299778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.299746 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-kube-api-access-6mbrt\" (UniqueName: \"kubernetes.io/projected/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-kube-api-access-6mbrt\") pod \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " Oct 31 13:15:28.299930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.299816 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-maas-api-service-ca\") pod \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\" (UID: \"56c5cc9f-9c47-4155-855f-e8b9ad1a876e\") " Oct 31 13:15:28.301124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.300208 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-maas-api-service-ca" pod "56c5cc9f-9c47-4155-855f-e8b9ad1a876e" (UID: "56c5cc9f-9c47-4155-855f-e8b9ad1a876e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:28.301124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.300479 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:28.302623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.302592 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-kube-api-access-6mbrt" pod "56c5cc9f-9c47-4155-855f-e8b9ad1a876e" (UID: "56c5cc9f-9c47-4155-855f-e8b9ad1a876e"). InnerVolumeSpecName "kube-api-access-6mbrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:28.303715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.303688 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890875-rwhnk"] Oct 31 13:15:28.375991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.375967 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:28.401585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.401555 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mbrt\" (UniqueName: \"kubernetes.io/projected/56c5cc9f-9c47-4155-855f-e8b9ad1a876e-kube-api-access-6mbrt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:28.413495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.413454 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:28.426594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.426568 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:28.517015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.516973 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a45b88-fafa-420f-a7ff-99abed7bd013-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a45b88-fafa-420f-a7ff-99abed7bd013-maas-api-service-ca\") pod \"03a45b88-fafa-420f-a7ff-99abed7bd013\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " Oct 31 13:15:28.517195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.517169 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a45b88-fafa-420f-a7ff-99abed7bd013-kube-api-access-p98qt\" (UniqueName: \"kubernetes.io/projected/03a45b88-fafa-420f-a7ff-99abed7bd013-kube-api-access-p98qt\") pod \"03a45b88-fafa-420f-a7ff-99abed7bd013\" (UID: \"03a45b88-fafa-420f-a7ff-99abed7bd013\") " Oct 31 13:15:28.518717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.518539 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a45b88-fafa-420f-a7ff-99abed7bd013-maas-api-service-ca" pod "03a45b88-fafa-420f-a7ff-99abed7bd013" (UID: "03a45b88-fafa-420f-a7ff-99abed7bd013"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:15:28.523400 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.523360 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a45b88-fafa-420f-a7ff-99abed7bd013-kube-api-access-p98qt" pod "03a45b88-fafa-420f-a7ff-99abed7bd013" (UID: "03a45b88-fafa-420f-a7ff-99abed7bd013"). InnerVolumeSpecName "kube-api-access-p98qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:15:28.619614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.619535 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p98qt\" (UniqueName: \"kubernetes.io/projected/03a45b88-fafa-420f-a7ff-99abed7bd013-kube-api-access-p98qt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:28.619614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.619567 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a45b88-fafa-420f-a7ff-99abed7bd013-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:15:28.683053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.683020 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" Oct 31 13:15:28.683553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.683047 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96" event={"ID":"56c5cc9f-9c47-4155-855f-e8b9ad1a876e","Type":"ContainerDied","Data":"afacf9ac07c63a3ab75b09d6604766c9872a8676b1c0fb383da72b016a0633ee"} Oct 31 13:15:28.683553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.683123 5074 scope.go:122] "RemoveContainer" containerID="b6549e28a0b17b36116747dd051d20e495008965c711e4474b3059d3f155df98" Oct 31 13:15:28.691714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.691681 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" event={"ID":"03a45b88-fafa-420f-a7ff-99abed7bd013","Type":"ContainerDied","Data":"cdc882db3b0a8bcfe0a03d6e35db6c1b05bae17beed553a0f9eb5e40c5708249"} Oct 31 13:15:28.691902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.691799 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d" Oct 31 13:15:28.702484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.702458 5074 scope.go:122] "RemoveContainer" containerID="9ebdd6273b2abf2cd82e1fd5a3fe65110cfd1adad83eb2cf0a1ad3633c61753d" Oct 31 13:15:28.751878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.751846 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9"] Oct 31 13:15:28.756208 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:28.756177 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7723da_6f6e_4f6b_ada0_76fb6af99085.slice/crio-8c11d04b53cc8bcd7e3a5ac388a5746674945049f9c002e88f64afe183c5ff8c WatchSource:0}: Error finding container 8c11d04b53cc8bcd7e3a5ac388a5746674945049f9c002e88f64afe183c5ff8c: Status 404 returned error can't find the container with id 8c11d04b53cc8bcd7e3a5ac388a5746674945049f9c002e88f64afe183c5ff8c Oct 31 13:15:28.759696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.756775 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96"] Oct 31 13:15:28.769103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.769080 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890875-c9m96"] Oct 31 13:15:28.779794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.779768 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8"] Oct 31 13:15:28.780311 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:28.780286 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d9c999_6423_4ed8_9b67_47eda9032560.slice/crio-0151ed835ea0d69ae6ca46062177b859be41a89074873c15d837d4c0556abc9a WatchSource:0}: Error finding container 0151ed835ea0d69ae6ca46062177b859be41a89074873c15d837d4c0556abc9a: Status 404 returned error can't find the container with id 0151ed835ea0d69ae6ca46062177b859be41a89074873c15d837d4c0556abc9a Oct 31 13:15:28.864656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.852501 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d"] Oct 31 13:15:28.866933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:28.866900 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890875-9lq2d"] Oct 31 13:15:29.715982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.715944 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" event={"ID":"c8d9c999-6423-4ed8-9b67-47eda9032560","Type":"ContainerStarted","Data":"d227eaac233f92c94637af79b642070761909dcedc5f6276c5118e208a0eda81"} Oct 31 13:15:29.716496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.716000 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" event={"ID":"c8d9c999-6423-4ed8-9b67-47eda9032560","Type":"ContainerStarted","Data":"0151ed835ea0d69ae6ca46062177b859be41a89074873c15d837d4c0556abc9a"} Oct 31 13:15:29.716496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.716032 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:29.719782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.719748 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" event={"ID":"4f7723da-6f6e-4f6b-ada0-76fb6af99085","Type":"ContainerStarted","Data":"795d67acbbac53dfa8cba1d3db3b992e59172b56a87f2a1aa81de919a80199a8"} Oct 31 13:15:29.719956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.719826 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" event={"ID":"4f7723da-6f6e-4f6b-ada0-76fb6af99085","Type":"ContainerStarted","Data":"8c11d04b53cc8bcd7e3a5ac388a5746674945049f9c002e88f64afe183c5ff8c"} Oct 31 13:15:29.720194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.720160 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:29.745014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.744971 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" podStartSLOduration=2.744958745 podStartE2EDuration="2.744958745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:29.744668452 +0000 UTC m=+2623.922386436" watchObservedRunningTime="2026-10-31 13:15:29.744958745 +0000 UTC m=+2623.922676727" Oct 31 13:15:29.782074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.782008 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" podStartSLOduration=2.781995436 podStartE2EDuration="2.781995436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:29.781499642 +0000 UTC m=+2623.959217626" watchObservedRunningTime="2026-10-31 13:15:29.781995436 +0000 UTC m=+2623.959713415" Oct 31 13:15:29.953995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.953951 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" path="/var/lib/kubelet/pods/03a45b88-fafa-420f-a7ff-99abed7bd013/volumes" Oct 31 13:15:29.954560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.954541 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095b3e51-cd04-44ef-9798-fe0e94c19bef" path="/var/lib/kubelet/pods/095b3e51-cd04-44ef-9798-fe0e94c19bef/volumes" Oct 31 13:15:29.955043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.955023 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28505a68-5e10-4461-a95e-5680564eb5b7" path="/var/lib/kubelet/pods/28505a68-5e10-4461-a95e-5680564eb5b7/volumes" Oct 31 13:15:29.955896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.955873 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af9234f-cf40-4883-8d91-c392be69d425" path="/var/lib/kubelet/pods/2af9234f-cf40-4883-8d91-c392be69d425/volumes" Oct 31 13:15:29.956338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.956321 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53dd0ac7-66bd-4841-aa0e-0a0b41294dd3" path="/var/lib/kubelet/pods/53dd0ac7-66bd-4841-aa0e-0a0b41294dd3/volumes" Oct 31 13:15:29.956811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.956786 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" path="/var/lib/kubelet/pods/56c5cc9f-9c47-4155-855f-e8b9ad1a876e/volumes" Oct 31 13:15:29.957496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.957476 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721a5252-6907-4590-a6fc-bd7e1dc2a780" path="/var/lib/kubelet/pods/721a5252-6907-4590-a6fc-bd7e1dc2a780/volumes" Oct 31 13:15:29.957908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.957892 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f9576c-b31b-4cb0-84b9-32a6c471ab16" path="/var/lib/kubelet/pods/d6f9576c-b31b-4cb0-84b9-32a6c471ab16/volumes" Oct 31 13:15:29.958316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:29.958303 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ac994b-957d-47e5-8c0d-20dbc4c2ea80" path="/var/lib/kubelet/pods/e1ac994b-957d-47e5-8c0d-20dbc4c2ea80/volumes" Oct 31 13:15:33.923112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.923072 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv"] Oct 31 13:15:33.924028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924003 5074 cpu_manager.go:414] "containerMap: removing container" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:33.924028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924027 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:33.924199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924055 5074 cpu_manager.go:414] "containerMap: removing container" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:33.924199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924063 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:33.924199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924111 5074 cpu_manager.go:414] "containerMap: removing container" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:33.924199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924120 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:33.924199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924162 5074 cpu_manager.go:414] "containerMap: removing container" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:33.924199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924170 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:33.924496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924374 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:33.924496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924390 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:33.924496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924405 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:33.924496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.924423 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:33.944285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.936568 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:33.952627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.952592 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"openshift-service-ca.crt\"" Oct 31 13:15:33.952863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.952841 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-e2e--7a720dc0-dockercfg-bsn8h\"" Oct 31 13:15:33.952977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.952958 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-15a6a2\"" Oct 31 13:15:33.953142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.953123 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-733a4fb02d51\"" Oct 31 13:15:33.953344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.953326 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-49713eea7cf3\"" Oct 31 13:15:33.953464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.953450 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"routing-overlay\"" Oct 31 13:15:33.953535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.953521 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"kube-root-ca.crt\"" Oct 31 13:15:33.984563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.983627 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:33.984563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.983784 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshdq\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-kube-api-access-kshdq\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:33.984563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.984013 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:33.984563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:33.984082 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.016701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.016667 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv"] Oct 31 13:15:34.085272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.085241 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.085449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.085331 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.085515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.085474 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshdq\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-kube-api-access-kshdq\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.085711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.085666 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.086146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.086121 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.086279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.086255 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.088158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.088133 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.118445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.118411 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshdq\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-kube-api-access-kshdq\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.317382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.317347 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.620684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.620618 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv"] Oct 31 13:15:34.621583 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:34.621555 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19aa02ce_937c_4e87_bb40_ca06228aee9b.slice/crio-3ca62d7e1dd775fddbad974157f10410c6355d4374d5282e5eb304f42df5be13 WatchSource:0}: Error finding container 3ca62d7e1dd775fddbad974157f10410c6355d4374d5282e5eb304f42df5be13: Status 404 returned error can't find the container with id 3ca62d7e1dd775fddbad974157f10410c6355d4374d5282e5eb304f42df5be13 Oct 31 13:15:34.791053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.791005 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" event={"ID":"19aa02ce-937c-4e87-bb40-ca06228aee9b","Type":"ContainerStarted","Data":"39737cdab83bc4e1f24899bb831494804e1b4720cf9df028236a0a6d34a53de9"} Oct 31 13:15:34.791365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.791330 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" event={"ID":"19aa02ce-937c-4e87-bb40-ca06228aee9b","Type":"ContainerStarted","Data":"3ca62d7e1dd775fddbad974157f10410c6355d4374d5282e5eb304f42df5be13"} Oct 31 13:15:34.791702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.791683 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:34.823556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:34.823497 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" podStartSLOduration=1.823481178 podStartE2EDuration="1.823481178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:34.8212064 +0000 UTC m=+2628.998924384" watchObservedRunningTime="2026-10-31 13:15:34.823481178 +0000 UTC m=+2629.001199161" Oct 31 13:15:40.730903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:40.730866 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:15:40.731306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:40.730953 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:15:43.001117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.001076 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v"] Oct 31 13:15:43.002230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.002208 5074 cpu_manager.go:414] "containerMap: removing container" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:43.002314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.002232 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:43.002314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.002260 5074 cpu_manager.go:414] "containerMap: removing container" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:43.002314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.002268 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:43.002435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.002423 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c5cc9f-9c47-4155-855f-e8b9ad1a876e" containerName="cleanup" Oct 31 13:15:43.002471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.002452 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a45b88-fafa-420f-a7ff-99abed7bd013" containerName="cleanup" Oct 31 13:15:43.006864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.006843 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.017577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.012465 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0c1fbb-openshift-default-dockercfg-bgnzh\"" Oct 31 13:15:43.018490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.018465 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v"] Oct 31 13:15:43.166989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.166952 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-credential-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.166989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.166996 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istiod-ca-cert\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.167271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.167124 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-certs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.167271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.167263 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48bs\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-kube-api-access-p48bs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.167375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.167328 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-podinfo\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.167429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.167383 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-data\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.167480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.167460 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.167596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.167578 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-envoy\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.167726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.167705 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-token\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.268723 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48bs\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-kube-api-access-p48bs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.268809 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-podinfo\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.268868 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-data\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.268896 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269001 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-envoy\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269089 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-token\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269131 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-credential-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269156 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istiod-ca-cert\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269233 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-certs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.269476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269434 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-data\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.270130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269707 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-credential-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.270130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.269762 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-socket\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.270230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.270126 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istiod-ca-cert\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.272575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.270351 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-certs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.272575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.272287 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-envoy\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.272733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.272619 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-podinfo\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.283189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.283157 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48bs\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-kube-api-access-p48bs\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.284077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.284053 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-token\") pod \"e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.330024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.329993 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:43.668105 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:15:43.668068 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330a70c4_05f0_4716_b8e4_9bdd9a1d1fdf.slice/crio-ccbc1ee76658d3225b5593756821f061a6f7effa49dd0606f1d0bec131d563f1 WatchSource:0}: Error finding container ccbc1ee76658d3225b5593756821f061a6f7effa49dd0606f1d0bec131d563f1: Status 404 returned error can't find the container with id ccbc1ee76658d3225b5593756821f061a6f7effa49dd0606f1d0bec131d563f1 Oct 31 13:15:43.668216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.668161 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v"] Oct 31 13:15:43.671621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.671581 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:43.671732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.671709 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:43.671775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.671760 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:15:43.900517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.900473 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" event={"ID":"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf","Type":"ContainerStarted","Data":"aaaf8c08c1d00e8c8e89baa3b6cb1f1f870cd6bd9c5ca866cfbf7bc468ab3191"} Oct 31 13:15:43.900730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.900542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" event={"ID":"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf","Type":"ContainerStarted","Data":"ccbc1ee76658d3225b5593756821f061a6f7effa49dd0606f1d0bec131d563f1"} Oct 31 13:15:43.930115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:43.929997 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podStartSLOduration=1.9299756829999999 podStartE2EDuration="1.929975683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:43.92687501 +0000 UTC m=+2638.104592994" watchObservedRunningTime="2026-10-31 13:15:43.929975683 +0000 UTC m=+2638.107693661" Oct 31 13:15:44.332776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:44.332737 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:44.333516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:44.333486 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:15:44.333855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:44.333557 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 13:15:45.330650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:45.330608 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:15:45.330848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:45.330712 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 13:15:45.804753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:45.804722 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:15:46.330526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:46.330490 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:15:46.330825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:46.330796 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 13:15:47.330507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:47.330471 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:15:47.330936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:47.330531 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 13:15:48.330791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:48.330750 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:15:48.331190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:48.330831 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 13:15:49.334186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:49.334149 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 31 13:15:49.334820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:49.334775 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 31 13:15:50.336444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:50.336409 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:50.336987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:50.336751 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:50.337624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:15:50.337604 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:15:55.986692 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:15:55.986610 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:16:21.001051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:21.001007 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:16:26.090164 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:16:26.090121 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:16:31.941735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:31.941696 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w"] Oct 31 13:16:31.947586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:31.947564 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:31.952687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:31.952655 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:16:31.953196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:31.953174 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:16:31.953919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:31.953248 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:16:31.954038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:31.953334 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:16:31.955967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:31.955939 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w"] Oct 31 13:16:32.025808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.025752 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7j29\" (UniqueName: \"kubernetes.io/projected/b03df400-9cc2-41a7-802d-df6728e18264-kube-api-access-t7j29\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.026043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.025936 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.026043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.026004 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.128226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.128176 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j29\" (UniqueName: \"kubernetes.io/projected/b03df400-9cc2-41a7-802d-df6728e18264-kube-api-access-t7j29\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.128439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.128302 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.128439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.128356 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.129181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.129152 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.131102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.131067 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.140860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.140831 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7j29\" (UniqueName: \"kubernetes.io/projected/b03df400-9cc2-41a7-802d-df6728e18264-kube-api-access-t7j29\") pod \"maas-api-revoke-sub-8ad1a6cf-6mt7w\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.266461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.266417 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:32.558430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.558392 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w"] Oct 31 13:16:32.559044 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:16:32.559018 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03df400_9cc2_41a7_802d_df6728e18264.slice/crio-eb486413b8c31dd3d874c34d9d4924de2bba66b6ab168e9bdeccff79b69932f1 WatchSource:0}: Error finding container eb486413b8c31dd3d874c34d9d4924de2bba66b6ab168e9bdeccff79b69932f1: Status 404 returned error can't find the container with id eb486413b8c31dd3d874c34d9d4924de2bba66b6ab168e9bdeccff79b69932f1 Oct 31 13:16:32.833721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:32.833591 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" event={"ID":"b03df400-9cc2-41a7-802d-df6728e18264","Type":"ContainerStarted","Data":"eb486413b8c31dd3d874c34d9d4924de2bba66b6ab168e9bdeccff79b69932f1"} Oct 31 13:16:33.846019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:33.845921 5074 generic.go:356] "Generic (PLEG): container finished" podID="b03df400-9cc2-41a7-802d-df6728e18264" containerID="c5062b315f2c3b228828fc05e90a1897c9211d29d96499474de6d59155909854" exitCode=0 Oct 31 13:16:33.846398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:33.846006 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" event={"ID":"b03df400-9cc2-41a7-802d-df6728e18264","Type":"ContainerDied","Data":"c5062b315f2c3b228828fc05e90a1897c9211d29d96499474de6d59155909854"} Oct 31 13:16:35.050972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.050947 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:35.185546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.185467 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03df400-9cc2-41a7-802d-df6728e18264-kube-api-access-t7j29\" (UniqueName: \"kubernetes.io/projected/b03df400-9cc2-41a7-802d-df6728e18264-kube-api-access-t7j29\") pod \"b03df400-9cc2-41a7-802d-df6728e18264\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " Oct 31 13:16:35.185546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.185539 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-xks-ca\") pod \"b03df400-9cc2-41a7-802d-df6728e18264\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " Oct 31 13:16:35.185867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.185670 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-openshift-ca\") pod \"b03df400-9cc2-41a7-802d-df6728e18264\" (UID: \"b03df400-9cc2-41a7-802d-df6728e18264\") " Oct 31 13:16:35.186088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.186048 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-openshift-ca" pod "b03df400-9cc2-41a7-802d-df6728e18264" (UID: "b03df400-9cc2-41a7-802d-df6728e18264"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:35.186408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.186386 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:35.187902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.187879 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-xks-ca" pod "b03df400-9cc2-41a7-802d-df6728e18264" (UID: "b03df400-9cc2-41a7-802d-df6728e18264"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:16:35.188128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.188111 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03df400-9cc2-41a7-802d-df6728e18264-kube-api-access-t7j29" pod "b03df400-9cc2-41a7-802d-df6728e18264" (UID: "b03df400-9cc2-41a7-802d-df6728e18264"). InnerVolumeSpecName "kube-api-access-t7j29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:35.287834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.287791 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7j29\" (UniqueName: \"kubernetes.io/projected/b03df400-9cc2-41a7-802d-df6728e18264-kube-api-access-t7j29\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:35.287834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.287825 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03df400-9cc2-41a7-802d-df6728e18264-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:35.863410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.863371 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" event={"ID":"b03df400-9cc2-41a7-802d-df6728e18264","Type":"ContainerDied","Data":"eb486413b8c31dd3d874c34d9d4924de2bba66b6ab168e9bdeccff79b69932f1"} Oct 31 13:16:35.863410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.863395 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w" Oct 31 13:16:35.863658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:35.863422 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb486413b8c31dd3d874c34d9d4924de2bba66b6ab168e9bdeccff79b69932f1" Oct 31 13:16:38.670505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:38.670463 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t"] Oct 31 13:16:40.692890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:40.692850 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv"] Oct 31 13:16:43.263239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.263197 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv"] Oct 31 13:16:43.671616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.671506 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" podUID="3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" containerName="payload-processing" containerID="cri-o://3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404" gracePeriod=25 Oct 31 13:16:43.969139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.969107 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:16:43.978536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.978504 5074 generic.go:356] "Generic (PLEG): container finished" podID="3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" containerID="3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404" exitCode=0 Oct 31 13:16:43.978536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.978543 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" Oct 31 13:16:43.978769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.978578 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" event={"ID":"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51","Type":"ContainerDied","Data":"3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404"} Oct 31 13:16:43.978769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.978659 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t" event={"ID":"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51","Type":"ContainerDied","Data":"fbec6c87e1224309c2ae80f8c0adba23dd50b03363aaf7c34ba198a0dc2abe0f"} Oct 31 13:16:43.978769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.978683 5074 scope.go:122] "RemoveContainer" containerID="3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404" Oct 31 13:16:43.987221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.987194 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-provider-credentials\") pod \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " Oct 31 13:16:43.987369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.987318 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-plugins-config-volume\") pod \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " Oct 31 13:16:43.987518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.987497 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-routing-overlay-volume\") pod \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " Oct 31 13:16:43.987604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.987555 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-kube-api-access-42tgh\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-kube-api-access-42tgh\") pod \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\" (UID: \"3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51\") " Oct 31 13:16:43.989798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.989077 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-routing-overlay-volume" pod "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" (UID: "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:43.989798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.989563 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-plugins-config-volume" pod "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" (UID: "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:43.990957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.990934 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-provider-credentials" pod "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" (UID: "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:43.991964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.991941 5074 scope.go:122] "RemoveContainer" containerID="3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404" Oct 31 13:16:43.992326 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:16:43.992278 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404\": container with ID starting with 3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404 not found: ID does not exist" containerID="3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404" Oct 31 13:16:43.992434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.992330 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404"} err="failed to get container status \"3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404\": rpc error: code = NotFound desc = could not find container \"3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404\": container with ID starting with 3fa8981c28138ee05cc3180cae9f9983a5d59f664de6368b0b5851e3b6a6b404 not found: ID does not exist" Oct 31 13:16:43.993462 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:43.993440 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-kube-api-access-42tgh" pod "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" (UID: "3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51"). InnerVolumeSpecName "kube-api-access-42tgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:44.090497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:44.090467 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:44.090497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:44.090496 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:44.090497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:44.090506 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:44.090802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:44.090515 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42tgh\" (UniqueName: \"kubernetes.io/projected/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51-kube-api-access-42tgh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:44.314919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:44.314874 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t"] Oct 31 13:16:44.325684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:44.324285 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4b9ql7t"] Oct 31 13:16:45.693721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:45.693672 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" podUID="19aa02ce-937c-4e87-bb40-ca06228aee9b" containerName="payload-processing" containerID="cri-o://39737cdab83bc4e1f24899bb831494804e1b4720cf9df028236a0a6d34a53de9" gracePeriod=25 Oct 31 13:16:45.801120 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:16:45.801052 5074 logging.go:55] [core] [Channel #2543 SubChannel #2544]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.235:50052", ServerName: "10.132.0.235:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.235:50052: connect: connection refused" Oct 31 13:16:46.042717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.042684 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" path="/var/lib/kubelet/pods/3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51/volumes" Oct 31 13:16:46.045458 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:16:46.045039 5074 info.go:109] Failed to get network devices: lstat /sys/class/net/3ca62d7e1dd775f: no such file or directory Oct 31 13:16:46.054529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.054493 5074 generic.go:356] "Generic (PLEG): container finished" podID="19aa02ce-937c-4e87-bb40-ca06228aee9b" containerID="39737cdab83bc4e1f24899bb831494804e1b4720cf9df028236a0a6d34a53de9" exitCode=0 Oct 31 13:16:46.055007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.054982 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" event={"ID":"19aa02ce-937c-4e87-bb40-ca06228aee9b","Type":"ContainerDied","Data":"39737cdab83bc4e1f24899bb831494804e1b4720cf9df028236a0a6d34a53de9"} Oct 31 13:16:46.195241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.195213 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:16:46.260908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.260875 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-routing-overlay-volume\") pod \"19aa02ce-937c-4e87-bb40-ca06228aee9b\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " Oct 31 13:16:46.261128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.260932 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-kube-api-access-kshdq\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-kube-api-access-kshdq\") pod \"19aa02ce-937c-4e87-bb40-ca06228aee9b\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " Oct 31 13:16:46.261128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.261100 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-plugins-config-volume\") pod \"19aa02ce-937c-4e87-bb40-ca06228aee9b\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " Oct 31 13:16:46.261267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.261204 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-routing-overlay-volume" pod "19aa02ce-937c-4e87-bb40-ca06228aee9b" (UID: "19aa02ce-937c-4e87-bb40-ca06228aee9b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:46.261331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.261287 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-provider-credentials\") pod \"19aa02ce-937c-4e87-bb40-ca06228aee9b\" (UID: \"19aa02ce-937c-4e87-bb40-ca06228aee9b\") " Oct 31 13:16:46.262066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.262022 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:46.270698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.270667 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-plugins-config-volume" pod "19aa02ce-937c-4e87-bb40-ca06228aee9b" (UID: "19aa02ce-937c-4e87-bb40-ca06228aee9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:46.272893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.272817 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-provider-credentials" pod "19aa02ce-937c-4e87-bb40-ca06228aee9b" (UID: "19aa02ce-937c-4e87-bb40-ca06228aee9b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:46.272893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.272858 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-kube-api-access-kshdq" pod "19aa02ce-937c-4e87-bb40-ca06228aee9b" (UID: "19aa02ce-937c-4e87-bb40-ca06228aee9b"). InnerVolumeSpecName "kube-api-access-kshdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:46.363210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.363118 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kshdq\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-kube-api-access-kshdq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:46.363210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.363154 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa02ce-937c-4e87-bb40-ca06228aee9b-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:46.363210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.363168 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19aa02ce-937c-4e87-bb40-ca06228aee9b-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:46.525850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.525821 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:16:46.533134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.533110 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:16:46.801039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:46.800992 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" podUID="19aa02ce-937c-4e87-bb40-ca06228aee9b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.235:50052\" within 1s: context deadline exceeded" Oct 31 13:16:47.064323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:47.064246 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" Oct 31 13:16:47.064470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:47.064242 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv" event={"ID":"19aa02ce-937c-4e87-bb40-ca06228aee9b","Type":"ContainerDied","Data":"3ca62d7e1dd775fddbad974157f10410c6355d4374d5282e5eb304f42df5be13"} Oct 31 13:16:47.064470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:47.064380 5074 scope.go:122] "RemoveContainer" containerID="39737cdab83bc4e1f24899bb831494804e1b4720cf9df028236a0a6d34a53de9" Oct 31 13:16:47.112263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:47.112218 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv"] Oct 31 13:16:47.117082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:47.117055 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-6b65l9mjv"] Oct 31 13:16:47.939825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:47.939786 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19aa02ce-937c-4e87-bb40-ca06228aee9b" path="/var/lib/kubelet/pods/19aa02ce-937c-4e87-bb40-ca06228aee9b/volumes" Oct 31 13:16:48.263779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.263735 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" podUID="c8c6d080-831c-498d-9d5c-6fa4c94870e5" containerName="payload-processing" containerID="cri-o://f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3" gracePeriod=25 Oct 31 13:16:48.563082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.563054 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:16:48.622044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.621985 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-routing-overlay-volume\") pod \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " Oct 31 13:16:48.622235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.622193 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-provider-credentials\") pod \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " Oct 31 13:16:48.622345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.622329 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-plugins-config-volume\") pod \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " Oct 31 13:16:48.622487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.622448 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-routing-overlay-volume" pod "c8c6d080-831c-498d-9d5c-6fa4c94870e5" (UID: "c8c6d080-831c-498d-9d5c-6fa4c94870e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:48.622599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.622532 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-kube-api-access-tzm8x\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-kube-api-access-tzm8x\") pod \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\" (UID: \"c8c6d080-831c-498d-9d5c-6fa4c94870e5\") " Oct 31 13:16:48.623529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.623111 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-plugins-config-volume" pod "c8c6d080-831c-498d-9d5c-6fa4c94870e5" (UID: "c8c6d080-831c-498d-9d5c-6fa4c94870e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:16:48.624776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.624598 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:48.624896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.624627 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c6d080-831c-498d-9d5c-6fa4c94870e5-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:48.703192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.703126 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-provider-credentials" pod "c8c6d080-831c-498d-9d5c-6fa4c94870e5" (UID: "c8c6d080-831c-498d-9d5c-6fa4c94870e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:48.708933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.708872 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-kube-api-access-tzm8x" pod "c8c6d080-831c-498d-9d5c-6fa4c94870e5" (UID: "c8c6d080-831c-498d-9d5c-6fa4c94870e5"). InnerVolumeSpecName "kube-api-access-tzm8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:16:48.726675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.726626 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzm8x\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-kube-api-access-tzm8x\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:48.726675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:48.726674 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8c6d080-831c-498d-9d5c-6fa4c94870e5-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:16:49.120959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.120796 5074 generic.go:356] "Generic (PLEG): container finished" podID="c8c6d080-831c-498d-9d5c-6fa4c94870e5" containerID="f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3" exitCode=0 Oct 31 13:16:49.121471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.120974 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" event={"ID":"c8c6d080-831c-498d-9d5c-6fa4c94870e5","Type":"ContainerDied","Data":"f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3"} Oct 31 13:16:49.121471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.121049 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" Oct 31 13:16:49.121471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.121064 5074 scope.go:122] "RemoveContainer" containerID="f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3" Oct 31 13:16:49.121471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.121050 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv" event={"ID":"c8c6d080-831c-498d-9d5c-6fa4c94870e5","Type":"ContainerDied","Data":"e6bf5a653ffce6f860ccaa29ad33174cf5f4ffce3e57c42e5fefbc52ec123e4e"} Oct 31 13:16:49.142383 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.142362 5074 scope.go:122] "RemoveContainer" containerID="f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3" Oct 31 13:16:49.144493 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:16:49.144300 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3\": container with ID starting with f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3 not found: ID does not exist" containerID="f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3" Oct 31 13:16:49.144493 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.144344 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3"} err="failed to get container status \"f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3\": rpc error: code = NotFound desc = could not find container \"f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3\": container with ID starting with f0fb9832e512bea1a2bf80182dfc631c056a6ffaca93be43a1cd1c459622b4f3 not found: ID does not exist" Oct 31 13:16:49.165851 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.165814 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv"] Oct 31 13:16:49.176145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.176110 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c2c98b/payload-processing-external-model-e2e-extmod-b-c2c98b-5c868xmxv"] Oct 31 13:16:49.487655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.485649 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt"] Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.494997 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c8c6d080-831c-498d-9d5c-6fa4c94870e5" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495034 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c6d080-831c-498d-9d5c-6fa4c94870e5" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495064 5074 cpu_manager.go:414] "containerMap: removing container" podUID="19aa02ce-937c-4e87-bb40-ca06228aee9b" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495072 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19aa02ce-937c-4e87-bb40-ca06228aee9b" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495104 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495114 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495166 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b03df400-9cc2-41a7-802d-df6728e18264" containerName="revoke-keys" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495176 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03df400-9cc2-41a7-802d-df6728e18264" containerName="revoke-keys" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495376 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c6d080-831c-498d-9d5c-6fa4c94870e5" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495390 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="19aa02ce-937c-4e87-bb40-ca06228aee9b" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495403 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d10a9ea-ccde-4cc8-83fb-8f5ed46a7e51" containerName="payload-processing" Oct 31 13:16:49.495660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.495419 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03df400-9cc2-41a7-802d-df6728e18264" containerName="revoke-keys" Oct 31 13:16:49.522655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.513825 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.548678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.532042 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"openshift-service-ca.crt\"" Oct 31 13:16:49.548678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.532185 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-e2e--e205175e-dockercfg-ssrqv\"" Oct 31 13:16:49.548678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.532246 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-26173d7f0173\"" Oct 31 13:16:49.548678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.532340 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"kube-root-ca.crt\"" Oct 31 13:16:49.548678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.532398 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-c4174febc8d5\"" Oct 31 13:16:49.548678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.532678 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-29e3a0\"" Oct 31 13:16:49.548678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.533352 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"routing-overlay\"" Oct 31 13:16:49.550041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.550012 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt"] Oct 31 13:16:49.646486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.620540 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl"] Oct 31 13:16:49.649007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.648950 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl"] Oct 31 13:16:49.649137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.649114 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.652851 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.652813 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.653249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.653227 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9947\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-kube-api-access-p9947\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.653439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.653424 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.653684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.653666 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.654626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.654601 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c2c98b-openshift-default-dockercfg-pcv65\"" Oct 31 13:16:49.755276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755163 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp669\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-kube-api-access-xp669\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.755276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755210 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-data\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.755512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755291 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.755512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755403 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-podinfo\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.755512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755458 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-certs\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.755862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755572 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-envoy\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.755862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755616 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.755981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755918 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.755981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.755966 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-credential-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.756151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.756108 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.756151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.756144 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9947\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-kube-api-access-p9947\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.756290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.756192 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.756290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.756243 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-token\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.756380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.756360 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istiod-ca-cert\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.756906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.756888 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.758564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.758535 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.768921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.768898 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9947\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-kube-api-access-p9947\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.857564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.857519 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-envoy\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.857766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.857582 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.857766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.857647 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-credential-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.857871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.857842 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-token\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.857922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.857880 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istiod-ca-cert\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.857922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.857911 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp669\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-kube-api-access-xp669\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.858008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.857954 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-data\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.858121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.858100 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-podinfo\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.858206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.858160 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-certs\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.859445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.858432 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-credential-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.859445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.858490 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-certs\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.859445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.858804 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-data\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.859445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.859269 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istiod-ca-cert\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.860377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.860347 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-socket\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.860696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.860673 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-envoy\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.862288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.862262 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-podinfo\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.882126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.881868 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:49.884033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.883999 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp669\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-kube-api-access-xp669\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.888217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.888190 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-token\") pod \"e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:49.945392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.945361 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c6d080-831c-498d-9d5c-6fa4c94870e5" path="/var/lib/kubelet/pods/c8c6d080-831c-498d-9d5c-6fa4c94870e5/volumes" Oct 31 13:16:49.967728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:49.967422 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:50.213435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:50.213400 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt"] Oct 31 13:16:50.219416 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:16:50.219380 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice/crio-ea54ec8362eec15458262ba322371ac8b9509d152dbf22b6afab7e27d83f811d WatchSource:0}: Error finding container ea54ec8362eec15458262ba322371ac8b9509d152dbf22b6afab7e27d83f811d: Status 404 returned error can't find the container with id ea54ec8362eec15458262ba322371ac8b9509d152dbf22b6afab7e27d83f811d Oct 31 13:16:50.306110 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:16:50.306067 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998eeb9a_fec6_49bc_8657_c88bf5eecdef.slice/crio-931415f934e4ca70df69e090804b0b65e089aa88beb4015bfb3ca612333c92bf WatchSource:0}: Error finding container 931415f934e4ca70df69e090804b0b65e089aa88beb4015bfb3ca612333c92bf: Status 404 returned error can't find the container with id 931415f934e4ca70df69e090804b0b65e089aa88beb4015bfb3ca612333c92bf Oct 31 13:16:50.308582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:50.308408 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl"] Oct 31 13:16:50.310134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:50.310095 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:50.310216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:50.310190 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:50.310256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:50.310235 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:16:51.143736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.143612 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" event={"ID":"47d2abba-74f9-4487-9fc7-f9b0bca247e8","Type":"ContainerStarted","Data":"581cdb040c4bf1921a88608acc977ab3c1b98ead788e5d82b53116447f97af52"} Oct 31 13:16:51.143736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.143696 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" event={"ID":"47d2abba-74f9-4487-9fc7-f9b0bca247e8","Type":"ContainerStarted","Data":"ea54ec8362eec15458262ba322371ac8b9509d152dbf22b6afab7e27d83f811d"} Oct 31 13:16:51.143736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.143729 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:16:51.145549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.145523 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" event={"ID":"998eeb9a-fec6-49bc-8657-c88bf5eecdef","Type":"ContainerStarted","Data":"8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e"} Oct 31 13:16:51.145754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.145563 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" event={"ID":"998eeb9a-fec6-49bc-8657-c88bf5eecdef","Type":"ContainerStarted","Data":"931415f934e4ca70df69e090804b0b65e089aa88beb4015bfb3ca612333c92bf"} Oct 31 13:16:51.167494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.167442 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" podStartSLOduration=2.167426933 podStartE2EDuration="2.167426933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:51.165476174 +0000 UTC m=+2705.343194158" watchObservedRunningTime="2026-10-31 13:16:51.167426933 +0000 UTC m=+2705.345144945" Oct 31 13:16:51.196560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.196497 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podStartSLOduration=2.196481803 podStartE2EDuration="2.196481803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:51.192907447 +0000 UTC m=+2705.370625430" watchObservedRunningTime="2026-10-31 13:16:51.196481803 +0000 UTC m=+2705.374199784" Oct 31 13:16:51.968513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.968478 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:51.989657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.988911 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:16:51.989657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:51.988979 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:16:52.972667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:52.968620 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:16:52.972667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:52.968730 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:16:53.968043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:53.968009 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:16:53.968238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:53.968062 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:16:54.991659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:54.981085 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:16:54.991659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:54.981159 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:16:55.980660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:55.968754 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:16:55.980660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:55.968817 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:16:56.014664 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:16:56.014183 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:16:57.044382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:57.044340 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:57.045800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:57.045773 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:16:57.047296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:16:57.047271 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:17:01.209844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:01.203431 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:17:02.180877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:02.180838 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:17:03.096809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:03.096772 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg"] Oct 31 13:17:03.112663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:03.111089 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4156835-c49vg"] Oct 31 13:17:03.941664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:03.941602 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6571282c-39cd-40ae-b384-4a3cf36a7a90" path="/var/lib/kubelet/pods/6571282c-39cd-40ae-b384-4a3cf36a7a90/volumes" Oct 31 13:17:13.137046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:13.137011 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf"] Oct 31 13:17:13.200272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:13.200236 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c54139a-tdsjf"] Oct 31 13:17:13.941159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:13.941118 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b166cb2-5866-4357-ac29-00be168f8a0e" path="/var/lib/kubelet/pods/0b166cb2-5866-4357-ac29-00be168f8a0e/volumes" Oct 31 13:17:19.771770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:19.771728 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:17:26.030220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.029926 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs"] Oct 31 13:17:26.038048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.038017 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs"] Oct 31 13:17:26.038315 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.038182 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.048578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.047887 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"routing-overlay\"" Oct 31 13:17:26.048578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.048178 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"openshift-service-ca.crt\"" Oct 31 13:17:26.048578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.048254 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"kube-root-ca.crt\"" Oct 31 13:17:26.048578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.048293 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-54a16b\"" Oct 31 13:17:26.080539 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:17:26.080502 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:17:26.083453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.083421 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-e2e--427d574a-dockercfg-7jw2f\"" Oct 31 13:17:26.083623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.083507 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-1b52f450f69f\"" Oct 31 13:17:26.125111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.125079 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.125345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.125246 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2f6\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-kube-api-access-rc2f6\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.125489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.125467 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.125816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.125791 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.240007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.239964 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2f6\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-kube-api-access-rc2f6\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.240418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.240394 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.240689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.240667 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.240954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.240936 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.242129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.242103 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.252518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.252481 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.282709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.268419 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.299612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.299573 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2f6\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-kube-api-access-rc2f6\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.407538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.405145 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:26.547257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.547179 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j"] Oct 31 13:17:26.553478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.553448 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.577039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.576992 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-dc4563f34129\"" Oct 31 13:17:26.581438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.578386 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-e4da4b834193\"" Oct 31 13:17:26.587086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.587060 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-5db32c229bfc\"" Oct 31 13:17:26.587086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.587073 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"openshift-service-ca.crt\"" Oct 31 13:17:26.587303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.587089 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"routing-overlay\"" Oct 31 13:17:26.587303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.587173 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"kube-root-ca.crt\"" Oct 31 13:17:26.587303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.587207 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb\"" Oct 31 13:17:26.587303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.587284 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-e2e--9527f114-dockercfg-5ljgf\"" Oct 31 13:17:26.590221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.590194 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j"] Oct 31 13:17:26.823736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.811726 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdz77\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-kube-api-access-xdz77\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.823736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.811892 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.823736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.812082 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.823736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.812196 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.981337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.917934 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.981337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.918078 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.981337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.918219 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdz77\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-kube-api-access-xdz77\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.981337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:26.918312 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:26.981337 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:26.950401 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960ab4a5_325e_4058_ae15_b261a8d76f9e.slice/crio-f3353f6db48118a423be7f2861cedecc5194018939e0211c3e80a5d4a8a3036f WatchSource:0}: Error finding container f3353f6db48118a423be7f2861cedecc5194018939e0211c3e80a5d4a8a3036f: Status 404 returned error can't find the container with id f3353f6db48118a423be7f2861cedecc5194018939e0211c3e80a5d4a8a3036f Oct 31 13:17:27.018774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.002352 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:27.028032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.027992 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs"] Oct 31 13:17:27.033334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.033171 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:27.036028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.035977 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:27.041848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.041820 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdz77\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-kube-api-access-xdz77\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:27.170837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.170799 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:27.289025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.288986 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72"] Oct 31 13:17:27.297192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.297162 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.316328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.316300 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-29e3a0\"/\"agc-cred-bd9afd006e3a\"" Oct 31 13:17:27.329470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.329441 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.329883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.329480 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4xl\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-kube-api-access-vd4xl\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.329883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.329695 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.330066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.329898 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.357202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.357123 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72"] Oct 31 13:17:27.431513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.431470 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.431753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.431729 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.431821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.431769 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4xl\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-kube-api-access-vd4xl\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.431870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.431854 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.432391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.432367 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.432495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.432369 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.434523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.434502 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.455859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.455826 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4xl\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-kube-api-access-vd4xl\") pod \"payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.499934 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:27.499903 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cbdbcb_fa6c_47b5_acd1_513d95fdeceb.slice/crio-2b1fd89184331bf6d486b5a1961851ac9692fb2dcb9f0fc67462fcd9d7646348 WatchSource:0}: Error finding container 2b1fd89184331bf6d486b5a1961851ac9692fb2dcb9f0fc67462fcd9d7646348: Status 404 returned error can't find the container with id 2b1fd89184331bf6d486b5a1961851ac9692fb2dcb9f0fc67462fcd9d7646348 Oct 31 13:17:27.500277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.500251 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j"] Oct 31 13:17:27.613280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.613221 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:27.944165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.944070 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" event={"ID":"960ab4a5-325e-4058-ae15-b261a8d76f9e","Type":"ContainerStarted","Data":"fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754"} Oct 31 13:17:27.944165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.944156 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" event={"ID":"960ab4a5-325e-4058-ae15-b261a8d76f9e","Type":"ContainerStarted","Data":"f3353f6db48118a423be7f2861cedecc5194018939e0211c3e80a5d4a8a3036f"} Oct 31 13:17:27.944357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.944183 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:27.944357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.944198 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" event={"ID":"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb","Type":"ContainerStarted","Data":"70fcae20ed532b77419f6f6df6af03417e6fad7b82d343409c52f105341a5c35"} Oct 31 13:17:27.944357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.944222 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:27.944357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.944242 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" event={"ID":"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb","Type":"ContainerStarted","Data":"2b1fd89184331bf6d486b5a1961851ac9692fb2dcb9f0fc67462fcd9d7646348"} Oct 31 13:17:27.951935 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.951899 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72"] Oct 31 13:17:27.956536 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:27.956506 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee85e49_07b8_4e0c_bb60_b9e7250b2214.slice/crio-3a4ccf09f4e84048a6d27cd1bdc0c50f4c358ddf46a901fb52575394650b883f WatchSource:0}: Error finding container 3a4ccf09f4e84048a6d27cd1bdc0c50f4c358ddf46a901fb52575394650b883f: Status 404 returned error can't find the container with id 3a4ccf09f4e84048a6d27cd1bdc0c50f4c358ddf46a901fb52575394650b883f Oct 31 13:17:27.970139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:27.970098 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" podStartSLOduration=2.970085519 podStartE2EDuration="2.970085519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:27.96815791 +0000 UTC m=+2742.145875906" watchObservedRunningTime="2026-10-31 13:17:27.970085519 +0000 UTC m=+2742.147803545" Oct 31 13:17:28.000571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:28.000311 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" podStartSLOduration=2.000295065 podStartE2EDuration="2.000295065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:27.999195603 +0000 UTC m=+2742.176913610" watchObservedRunningTime="2026-10-31 13:17:28.000295065 +0000 UTC m=+2742.178013052" Oct 31 13:17:28.946785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:28.946747 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" event={"ID":"eee85e49-07b8-4e0c-bb60-b9e7250b2214","Type":"ContainerStarted","Data":"4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606"} Oct 31 13:17:28.947323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:28.946815 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" event={"ID":"eee85e49-07b8-4e0c-bb60-b9e7250b2214","Type":"ContainerStarted","Data":"3a4ccf09f4e84048a6d27cd1bdc0c50f4c358ddf46a901fb52575394650b883f"} Oct 31 13:17:28.947323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:28.946843 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:28.974301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:28.974250 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" podStartSLOduration=1.974237162 podStartE2EDuration="1.974237162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:28.970541492 +0000 UTC m=+2743.148259477" watchObservedRunningTime="2026-10-31 13:17:28.974237162 +0000 UTC m=+2743.151955169" Oct 31 13:17:32.011776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:32.011726 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs"] Oct 31 13:17:32.028585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:32.028265 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:32.908655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:32.905818 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j"] Oct 31 13:17:32.919656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:32.913375 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:34.425554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.425517 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr"] Oct 31 13:17:34.430296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.430278 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.440519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.440097 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:17:34.440519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.440182 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:17:34.441660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.441606 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:17:34.441765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.441628 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:17:34.441946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.441926 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr"] Oct 31 13:17:34.498799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.498746 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.499007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.498830 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbj9\" (UniqueName: \"kubernetes.io/projected/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-kube-api-access-hvbj9\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.499155 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.499115 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.600892 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.600855 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.601099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.600907 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbj9\" (UniqueName: \"kubernetes.io/projected/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-kube-api-access-hvbj9\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.601099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.601063 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.601964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.601935 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.603357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.603329 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.613933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.613901 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbj9\" (UniqueName: \"kubernetes.io/projected/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-kube-api-access-hvbj9\") pod \"maas-api-revoke-sub-23b63abe-jv2xr\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:34.748215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:34.748179 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:35.083586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:35.082612 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr"] Oct 31 13:17:35.130076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:35.130045 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" event={"ID":"ff8e1ebd-c9e0-497b-b941-b7d64edc141e","Type":"ContainerStarted","Data":"868cbda0f095d2995c6b0475726999573c0f2cd8b53eb97a149af40234de358e"} Oct 31 13:17:36.140151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:36.140114 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" event={"ID":"ff8e1ebd-c9e0-497b-b941-b7d64edc141e","Type":"ContainerStarted","Data":"9ff3f9a617d1f9493094483d058dd107f56e7ef03f6e2fb49f2872453877c1ae"} Oct 31 13:17:36.160380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:36.160333 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" podStartSLOduration=2.160320336 podStartE2EDuration="2.160320336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:36.159564504 +0000 UTC m=+2750.337282509" watchObservedRunningTime="2026-10-31 13:17:36.160320336 +0000 UTC m=+2750.338038320" Oct 31 13:17:37.012967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.012901 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" podUID="960ab4a5-325e-4058-ae15-b261a8d76f9e" containerName="payload-processing" containerID="cri-o://fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754" gracePeriod=25 Oct 31 13:17:37.227661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.227556 5074 generic.go:356] "Generic (PLEG): container finished" podID="ff8e1ebd-c9e0-497b-b941-b7d64edc141e" containerID="9ff3f9a617d1f9493094483d058dd107f56e7ef03f6e2fb49f2872453877c1ae" exitCode=0 Oct 31 13:17:37.240177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.239611 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" event={"ID":"ff8e1ebd-c9e0-497b-b941-b7d64edc141e","Type":"ContainerDied","Data":"9ff3f9a617d1f9493094483d058dd107f56e7ef03f6e2fb49f2872453877c1ae"} Oct 31 13:17:37.527737 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.527710 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:37.678608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.678515 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-kube-api-access-rc2f6\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-kube-api-access-rc2f6\") pod \"960ab4a5-325e-4058-ae15-b261a8d76f9e\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " Oct 31 13:17:37.678837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.678625 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-provider-credentials\") pod \"960ab4a5-325e-4058-ae15-b261a8d76f9e\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " Oct 31 13:17:37.678837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.678768 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-routing-overlay-volume\") pod \"960ab4a5-325e-4058-ae15-b261a8d76f9e\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " Oct 31 13:17:37.678964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.678837 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-plugins-config-volume\") pod \"960ab4a5-325e-4058-ae15-b261a8d76f9e\" (UID: \"960ab4a5-325e-4058-ae15-b261a8d76f9e\") " Oct 31 13:17:37.679175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.679148 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-routing-overlay-volume" pod "960ab4a5-325e-4058-ae15-b261a8d76f9e" (UID: "960ab4a5-325e-4058-ae15-b261a8d76f9e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:37.679297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.679266 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-plugins-config-volume" pod "960ab4a5-325e-4058-ae15-b261a8d76f9e" (UID: "960ab4a5-325e-4058-ae15-b261a8d76f9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:37.679747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.679720 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:37.679873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.679751 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960ab4a5-325e-4058-ae15-b261a8d76f9e-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:37.681026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.681002 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-provider-credentials" pod "960ab4a5-325e-4058-ae15-b261a8d76f9e" (UID: "960ab4a5-325e-4058-ae15-b261a8d76f9e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:37.681192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.681176 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-kube-api-access-rc2f6" pod "960ab4a5-325e-4058-ae15-b261a8d76f9e" (UID: "960ab4a5-325e-4058-ae15-b261a8d76f9e"). InnerVolumeSpecName "kube-api-access-rc2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:37.781276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.781240 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2f6\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-kube-api-access-rc2f6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:37.781276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.781272 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/960ab4a5-325e-4058-ae15-b261a8d76f9e-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:37.906297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:37.906246 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" podUID="f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" containerName="payload-processing" containerID="cri-o://70fcae20ed532b77419f6f6df6af03417e6fad7b82d343409c52f105341a5c35" gracePeriod=25 Oct 31 13:17:38.318754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.318418 5074 generic.go:356] "Generic (PLEG): container finished" podID="f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" containerID="70fcae20ed532b77419f6f6df6af03417e6fad7b82d343409c52f105341a5c35" exitCode=0 Oct 31 13:17:38.318754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.318672 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" event={"ID":"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb","Type":"ContainerDied","Data":"70fcae20ed532b77419f6f6df6af03417e6fad7b82d343409c52f105341a5c35"} Oct 31 13:17:38.321654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.321038 5074 generic.go:356] "Generic (PLEG): container finished" podID="960ab4a5-325e-4058-ae15-b261a8d76f9e" containerID="fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754" exitCode=0 Oct 31 13:17:38.321654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.321333 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" Oct 31 13:17:38.322725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.322267 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" event={"ID":"960ab4a5-325e-4058-ae15-b261a8d76f9e","Type":"ContainerDied","Data":"fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754"} Oct 31 13:17:38.322725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.322316 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs" event={"ID":"960ab4a5-325e-4058-ae15-b261a8d76f9e","Type":"ContainerDied","Data":"f3353f6db48118a423be7f2861cedecc5194018939e0211c3e80a5d4a8a3036f"} Oct 31 13:17:38.322725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.322346 5074 scope.go:122] "RemoveContainer" containerID="fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754" Oct 31 13:17:38.389746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.389705 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs"] Oct 31 13:17:38.397446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.397412 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-56d8njdvs"] Oct 31 13:17:38.445061 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.440316 5074 scope.go:122] "RemoveContainer" containerID="fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754" Oct 31 13:17:38.445554 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:17:38.445208 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754\": container with ID starting with fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754 not found: ID does not exist" containerID="fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754" Oct 31 13:17:38.445554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.445251 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754"} err="failed to get container status \"fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754\": rpc error: code = NotFound desc = could not find container \"fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754\": container with ID starting with fc2b964037c7568340d2b60e9951f7968a4c5fcbd135aab642f6607c4ff9d754 not found: ID does not exist" Oct 31 13:17:38.623227 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.623198 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:38.671998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.671619 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:38.749685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.749651 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-provider-credentials\") pod \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " Oct 31 13:17:38.749974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.749948 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-routing-overlay-volume\") pod \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " Oct 31 13:17:38.750093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.750029 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-kube-api-access-xdz77\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-kube-api-access-xdz77\") pod \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " Oct 31 13:17:38.750093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.750083 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-plugins-config-volume\") pod \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\" (UID: \"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb\") " Oct 31 13:17:38.750829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.750789 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-plugins-config-volume" pod "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" (UID: "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:38.751014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.750968 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:38.752804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.752764 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-provider-credentials" pod "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" (UID: "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:38.753115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.753091 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-routing-overlay-volume" pod "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" (UID: "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:38.753491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.753457 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-kube-api-access-xdz77" pod "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" (UID: "f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb"). InnerVolumeSpecName "kube-api-access-xdz77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:38.852094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852014 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-kube-api-access-hvbj9\" (UniqueName: \"kubernetes.io/projected/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-kube-api-access-hvbj9\") pod \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " Oct 31 13:17:38.852094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852064 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-xks-ca\") pod \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " Oct 31 13:17:38.852344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852145 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-openshift-ca\") pod \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\" (UID: \"ff8e1ebd-c9e0-497b-b941-b7d64edc141e\") " Oct 31 13:17:38.852508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852479 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-openshift-ca" pod "ff8e1ebd-c9e0-497b-b941-b7d64edc141e" (UID: "ff8e1ebd-c9e0-497b-b941-b7d64edc141e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:38.852777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852751 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:38.852903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852779 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:38.852903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852792 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:38.852903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.852801 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdz77\" (UniqueName: \"kubernetes.io/projected/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb-kube-api-access-xdz77\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:38.854172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.854146 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-kube-api-access-hvbj9" pod "ff8e1ebd-c9e0-497b-b941-b7d64edc141e" (UID: "ff8e1ebd-c9e0-497b-b941-b7d64edc141e"). InnerVolumeSpecName "kube-api-access-hvbj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:38.854255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.854192 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-xks-ca" pod "ff8e1ebd-c9e0-497b-b941-b7d64edc141e" (UID: "ff8e1ebd-c9e0-497b-b941-b7d64edc141e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:17:38.870684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.870656 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt"] Oct 31 13:17:38.871739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.871718 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" containerName="payload-processing" Oct 31 13:17:38.871739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.871739 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" containerName="payload-processing" Oct 31 13:17:38.871911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.871776 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ff8e1ebd-c9e0-497b-b941-b7d64edc141e" containerName="revoke-keys" Oct 31 13:17:38.871911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.871786 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8e1ebd-c9e0-497b-b941-b7d64edc141e" containerName="revoke-keys" Oct 31 13:17:38.871911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.871807 5074 cpu_manager.go:414] "containerMap: removing container" podUID="960ab4a5-325e-4058-ae15-b261a8d76f9e" containerName="payload-processing" Oct 31 13:17:38.871911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.871814 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960ab4a5-325e-4058-ae15-b261a8d76f9e" containerName="payload-processing" Oct 31 13:17:38.872118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.872033 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8e1ebd-c9e0-497b-b941-b7d64edc141e" containerName="revoke-keys" Oct 31 13:17:38.872118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.872049 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="960ab4a5-325e-4058-ae15-b261a8d76f9e" containerName="payload-processing" Oct 31 13:17:38.872118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.872070 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" containerName="payload-processing" Oct 31 13:17:38.876958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.876938 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:38.889247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.888907 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"routing-overlay\"" Oct 31 13:17:38.889247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.888937 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"kube-root-ca.crt\"" Oct 31 13:17:38.889247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.888984 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"agc-cred-811c6a0d5a4c\"" Oct 31 13:17:38.889247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.889009 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"openshift-service-ca.crt\"" Oct 31 13:17:38.889247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.889071 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2ef686\"" Oct 31 13:17:38.889247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.889114 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-e2e--362fde4e-dockercfg-l7p2q\"" Oct 31 13:17:38.892352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.892330 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt"] Oct 31 13:17:38.953952 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.953919 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:38.954151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.954125 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:38.954223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.954178 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:38.954285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.954230 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpm5\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-kube-api-access-9rpm5\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:38.954355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.954316 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvbj9\" (UniqueName: \"kubernetes.io/projected/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-kube-api-access-hvbj9\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:38.954355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:38.954327 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff8e1ebd-c9e0-497b-b941-b7d64edc141e-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:39.064657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.058397 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.064657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.058496 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.064657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.058575 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpm5\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-kube-api-access-9rpm5\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.064657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.058702 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.082655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.077653 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.082655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.078450 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.130703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.121483 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpm5\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-kube-api-access-9rpm5\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.152660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.142999 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.197653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.197112 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:39.396263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.395039 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" event={"ID":"f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb","Type":"ContainerDied","Data":"2b1fd89184331bf6d486b5a1961851ac9692fb2dcb9f0fc67462fcd9d7646348"} Oct 31 13:17:39.396263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.395113 5074 scope.go:122] "RemoveContainer" containerID="70fcae20ed532b77419f6f6df6af03417e6fad7b82d343409c52f105341a5c35" Oct 31 13:17:39.396263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.395396 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j" Oct 31 13:17:39.411671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.411477 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" event={"ID":"ff8e1ebd-c9e0-497b-b941-b7d64edc141e","Type":"ContainerDied","Data":"868cbda0f095d2995c6b0475726999573c0f2cd8b53eb97a149af40234de358e"} Oct 31 13:17:39.411671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.411554 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868cbda0f095d2995c6b0475726999573c0f2cd8b53eb97a149af40234de358e" Oct 31 13:17:39.413019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.412721 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr" Oct 31 13:17:39.455314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.455191 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j"] Oct 31 13:17:39.461292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.461158 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-6984jxn7j"] Oct 31 13:17:39.689731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.689643 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt"] Oct 31 13:17:39.693726 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:39.693692 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7949dc6f_fd7b_445d_9177_0aa1635e8a00.slice/crio-b33882bbbb85778fc6e22dddf5965393af5a4d18a6bcdcfd36a3e6cfb1991f86 WatchSource:0}: Error finding container b33882bbbb85778fc6e22dddf5965393af5a4d18a6bcdcfd36a3e6cfb1991f86: Status 404 returned error can't find the container with id b33882bbbb85778fc6e22dddf5965393af5a4d18a6bcdcfd36a3e6cfb1991f86 Oct 31 13:17:39.953754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.953673 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960ab4a5-325e-4058-ae15-b261a8d76f9e" path="/var/lib/kubelet/pods/960ab4a5-325e-4058-ae15-b261a8d76f9e/volumes" Oct 31 13:17:39.954195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.954180 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb" path="/var/lib/kubelet/pods/f8cbdbcb-fa6c-47b5-acd1-513d95fdeceb/volumes" Oct 31 13:17:39.959072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:39.959049 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:17:40.019662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:40.019137 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt"] Oct 31 13:17:40.431082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:40.431037 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" event={"ID":"7949dc6f-fd7b-445d-9177-0aa1635e8a00","Type":"ContainerStarted","Data":"90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f"} Oct 31 13:17:40.431522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:40.431111 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" event={"ID":"7949dc6f-fd7b-445d-9177-0aa1635e8a00","Type":"ContainerStarted","Data":"b33882bbbb85778fc6e22dddf5965393af5a4d18a6bcdcfd36a3e6cfb1991f86"} Oct 31 13:17:40.431522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:40.431141 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:40.474934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:40.472888 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" podStartSLOduration=2.472867937 podStartE2EDuration="2.472867937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:40.463301299 +0000 UTC m=+2754.641019282" watchObservedRunningTime="2026-10-31 13:17:40.472867937 +0000 UTC m=+2754.650585924" Oct 31 13:17:43.027002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:43.026962 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9"] Oct 31 13:17:43.031871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:43.031840 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf"] Oct 31 13:17:43.035389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:43.035370 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8657a3-g22r9"] Oct 31 13:17:43.038922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:43.038900 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a32386-mc9pf"] Oct 31 13:17:43.941073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:43.941032 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f4ddd7-ccbb-4858-87c3-f7b69dbe9070" path="/var/lib/kubelet/pods/25f4ddd7-ccbb-4858-87c3-f7b69dbe9070/volumes" Oct 31 13:17:43.941626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:43.941602 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e268dce8-438c-4f50-b0ad-7253f98a5eeb" path="/var/lib/kubelet/pods/e268dce8-438c-4f50-b0ad-7253f98a5eeb/volumes" Oct 31 13:17:45.021514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.020385 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" podUID="47d2abba-74f9-4487-9fc7-f9b0bca247e8" containerName="payload-processing" containerID="cri-o://581cdb040c4bf1921a88608acc977ab3c1b98ead788e5d82b53116447f97af52" gracePeriod=25 Oct 31 13:17:45.040785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.040702 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr"] Oct 31 13:17:45.043776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.043747 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68946767-7nwbr"] Oct 31 13:17:45.524789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.524752 5074 generic.go:356] "Generic (PLEG): container finished" podID="47d2abba-74f9-4487-9fc7-f9b0bca247e8" containerID="581cdb040c4bf1921a88608acc977ab3c1b98ead788e5d82b53116447f97af52" exitCode=0 Oct 31 13:17:45.525319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.525031 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" event={"ID":"47d2abba-74f9-4487-9fc7-f9b0bca247e8","Type":"ContainerDied","Data":"581cdb040c4bf1921a88608acc977ab3c1b98ead788e5d82b53116447f97af52"} Oct 31 13:17:45.578731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.546948 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:17:45.634657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.624853 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-kube-api-access-p9947\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-kube-api-access-p9947\") pod \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " Oct 31 13:17:45.634657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.625591 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-plugins-config-volume\") pod \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " Oct 31 13:17:45.634657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.625758 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-routing-overlay-volume\") pod \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " Oct 31 13:17:45.634657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.625793 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-provider-credentials\") pod \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\" (UID: \"47d2abba-74f9-4487-9fc7-f9b0bca247e8\") " Oct 31 13:17:45.634657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.627051 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-routing-overlay-volume" pod "47d2abba-74f9-4487-9fc7-f9b0bca247e8" (UID: "47d2abba-74f9-4487-9fc7-f9b0bca247e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:45.634657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.632186 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-plugins-config-volume" pod "47d2abba-74f9-4487-9fc7-f9b0bca247e8" (UID: "47d2abba-74f9-4487-9fc7-f9b0bca247e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:45.652744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.648345 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-provider-credentials" pod "47d2abba-74f9-4487-9fc7-f9b0bca247e8" (UID: "47d2abba-74f9-4487-9fc7-f9b0bca247e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:45.660889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.660850 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-kube-api-access-p9947" pod "47d2abba-74f9-4487-9fc7-f9b0bca247e8" (UID: "47d2abba-74f9-4487-9fc7-f9b0bca247e8"). InnerVolumeSpecName "kube-api-access-p9947". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:45.728261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.728226 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9947\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-kube-api-access-p9947\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:45.728261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.728259 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:45.728460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.728278 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47d2abba-74f9-4487-9fc7-f9b0bca247e8-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:45.728460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:45.728287 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47d2abba-74f9-4487-9fc7-f9b0bca247e8-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:46.020700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.018291 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00af11da-a413-405d-be46-651ae8accff3" path="/var/lib/kubelet/pods/00af11da-a413-405d-be46-651ae8accff3/volumes" Oct 31 13:17:46.095825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.094021 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6"] Oct 31 13:17:46.095825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.095155 5074 cpu_manager.go:414] "containerMap: removing container" podUID="47d2abba-74f9-4487-9fc7-f9b0bca247e8" containerName="payload-processing" Oct 31 13:17:46.095825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.095179 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d2abba-74f9-4487-9fc7-f9b0bca247e8" containerName="payload-processing" Oct 31 13:17:46.111993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.111339 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d2abba-74f9-4487-9fc7-f9b0bca247e8" containerName="payload-processing" Oct 31 13:17:46.133208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.125977 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.149117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.140620 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0c1fbb\"" Oct 31 13:17:46.149117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.140776 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"openshift-service-ca.crt\"" Oct 31 13:17:46.149117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.140851 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"kube-root-ca.crt\"" Oct 31 13:17:46.149117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.140926 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"routing-overlay\"" Oct 31 13:17:46.180694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.169033 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"payload-processing-external-model-e2e--9527f114-dockercfg-5ljgf\"" Oct 31 13:17:46.185761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.171004 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.186067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.186039 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.191962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.186263 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp5k\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-kube-api-access-bsp5k\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.192341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.192316 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.192840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.171496 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0c1fbb\"/\"agc-cred-e4da4b834193\"" Oct 31 13:17:46.227110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.227080 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6"] Oct 31 13:17:46.316623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.314906 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.316623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.315018 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.316623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.315077 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp5k\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-kube-api-access-bsp5k\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.316623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.315222 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.316623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.316346 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.337581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.322066 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.384236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.384186 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp5k\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-kube-api-access-bsp5k\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.394292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.390652 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm"] Oct 31 13:17:46.444662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.427101 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.444662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.429927 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.466316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.466265 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm"] Oct 31 13:17:46.475081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.475038 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:17:46.476031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.475682 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:17:46.476031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.475763 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:17:46.489674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.478908 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:17:46.560654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.557077 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.560654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.557133 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.560654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.557255 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsgm\" (UniqueName: \"kubernetes.io/projected/7276306b-3f0b-49d8-8c54-878c666ac783-kube-api-access-2xsgm\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.578050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.577720 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" event={"ID":"47d2abba-74f9-4487-9fc7-f9b0bca247e8","Type":"ContainerDied","Data":"ea54ec8362eec15458262ba322371ac8b9509d152dbf22b6afab7e27d83f811d"} Oct 31 13:17:46.578050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.577807 5074 scope.go:122] "RemoveContainer" containerID="581cdb040c4bf1921a88608acc977ab3c1b98ead788e5d82b53116447f97af52" Oct 31 13:17:46.578050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.577817 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt" Oct 31 13:17:46.620445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.620411 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt"] Oct 31 13:17:46.625956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.625927 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-df68w79nt"] Oct 31 13:17:46.659165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.659134 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.659341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.659201 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.659341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.659334 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsgm\" (UniqueName: \"kubernetes.io/projected/7276306b-3f0b-49d8-8c54-878c666ac783-kube-api-access-2xsgm\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.659907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.659882 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.661670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.661602 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.674236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.674207 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsgm\" (UniqueName: \"kubernetes.io/projected/7276306b-3f0b-49d8-8c54-878c666ac783-kube-api-access-2xsgm\") pod \"maas-api-revoke-sub-9e0cbc36-fz4rm\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:46.708257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.708227 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:46.859985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:46.859900 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:47.006718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.006678 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6"] Oct 31 13:17:47.007368 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:47.007332 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730af042_bf20_4386_a509_cccd48fb9df1.slice/crio-90a2d6cf07bab3534c4665df9c2821c3518530fe77c88cc2cc7a9c1c17f4ec17 WatchSource:0}: Error finding container 90a2d6cf07bab3534c4665df9c2821c3518530fe77c88cc2cc7a9c1c17f4ec17: Status 404 returned error can't find the container with id 90a2d6cf07bab3534c4665df9c2821c3518530fe77c88cc2cc7a9c1c17f4ec17 Oct 31 13:17:47.195092 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:47.195057 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7276306b_3f0b_49d8_8c54_878c666ac783.slice/crio-7a1e0e6dab005224c695ea921f3d955147d730d5bee68bbdc6eb236e3a657846 WatchSource:0}: Error finding container 7a1e0e6dab005224c695ea921f3d955147d730d5bee68bbdc6eb236e3a657846: Status 404 returned error can't find the container with id 7a1e0e6dab005224c695ea921f3d955147d730d5bee68bbdc6eb236e3a657846 Oct 31 13:17:47.198072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.198039 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm"] Oct 31 13:17:47.686713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.685975 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz"] Oct 31 13:17:47.721676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.715925 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.727039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.726982 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" event={"ID":"7276306b-3f0b-49d8-8c54-878c666ac783","Type":"ContainerStarted","Data":"7a1e0e6dab005224c695ea921f3d955147d730d5bee68bbdc6eb236e3a657846"} Oct 31 13:17:47.727307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.727264 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" event={"ID":"730af042-bf20-4386-a509-cccd48fb9df1","Type":"ContainerStarted","Data":"5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613"} Oct 31 13:17:47.727553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.727514 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:47.727726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.727707 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" event={"ID":"730af042-bf20-4386-a509-cccd48fb9df1","Type":"ContainerStarted","Data":"90a2d6cf07bab3534c4665df9c2821c3518530fe77c88cc2cc7a9c1c17f4ec17"} Oct 31 13:17:47.745931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.739431 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-15a6a2\"" Oct 31 13:17:47.745931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.739575 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-e2e--7a720dc0-dockercfg-bsn8h\"" Oct 31 13:17:47.757774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.752587 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"routing-overlay\"" Oct 31 13:17:47.757774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.752725 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-49713eea7cf3\"" Oct 31 13:17:47.757774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.752844 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"kube-root-ca.crt\"" Oct 31 13:17:47.757774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.752925 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"openshift-service-ca.crt\"" Oct 31 13:17:47.827942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.824781 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz"] Oct 31 13:17:47.830655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.830233 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t28jc\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-kube-api-access-t28jc\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.830655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.830288 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.830655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.830486 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.830957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.830672 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.932098 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.932060 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.932282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.932212 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t28jc\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-kube-api-access-t28jc\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.932381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.932262 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.932499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.932475 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.933373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.933346 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.933563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.933529 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.935413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.935390 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.945985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.945924 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t28jc\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-kube-api-access-t28jc\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:47.947175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:47.947149 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d2abba-74f9-4487-9fc7-f9b0bca247e8" path="/var/lib/kubelet/pods/47d2abba-74f9-4487-9fc7-f9b0bca247e8/volumes" Oct 31 13:17:48.067525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.067485 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:48.387657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.383685 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" podStartSLOduration=2.383662228 podStartE2EDuration="2.383662228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:47.864413138 +0000 UTC m=+2762.042131122" watchObservedRunningTime="2026-10-31 13:17:48.383662228 +0000 UTC m=+2762.561380212" Oct 31 13:17:48.400801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.400693 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz"] Oct 31 13:17:48.770068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.770034 5074 generic.go:356] "Generic (PLEG): container finished" podID="7276306b-3f0b-49d8-8c54-878c666ac783" containerID="a66a62a57fb8cc81b62eb87ed78487f335020459e00710f5280ca29ebbd8d24a" exitCode=0 Oct 31 13:17:48.770249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.770124 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" event={"ID":"7276306b-3f0b-49d8-8c54-878c666ac783","Type":"ContainerDied","Data":"a66a62a57fb8cc81b62eb87ed78487f335020459e00710f5280ca29ebbd8d24a"} Oct 31 13:17:48.772127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.772099 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" event={"ID":"bd08ad70-09bd-4903-9b55-8145aab7800e","Type":"ContainerStarted","Data":"0264697b7fb4c4d79cfdb065d5550f642995a3eee1451fd45d042a077d7d0ad3"} Oct 31 13:17:48.772237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.772154 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" event={"ID":"bd08ad70-09bd-4903-9b55-8145aab7800e","Type":"ContainerStarted","Data":"87c7af2883886b736d59c818f1978645931f2b304ff92a7645ee6b076058e75f"} Oct 31 13:17:48.772307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.772249 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:48.813645 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:48.813532 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" podStartSLOduration=1.813521309 podStartE2EDuration="1.813521309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:48.80987589 +0000 UTC m=+2762.987593875" watchObservedRunningTime="2026-10-31 13:17:48.813521309 +0000 UTC m=+2762.991239337" Oct 31 13:17:49.946027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.945986 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn"] Oct 31 13:17:49.951279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.951261 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:49.957702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.957681 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-15a6a2-openshift-default-dockercfg-kg58k\"" Oct 31 13:17:49.967185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.967156 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn"] Oct 31 13:17:49.971591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.971572 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:49.999793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.999756 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqn2\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-kube-api-access-dlqn2\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:49.999937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.999813 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-credential-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:49.999937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.999843 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istiod-ca-cert\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:49.999937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.999907 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-data\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.000118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:49.999946 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-certs\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.000118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.000078 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.000359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.000341 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-podinfo\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.000435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.000376 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-token\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.000495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.000437 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-envoy\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.101114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101079 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7276306b-3f0b-49d8-8c54-878c666ac783-kube-api-access-2xsgm\" (UniqueName: \"kubernetes.io/projected/7276306b-3f0b-49d8-8c54-878c666ac783-kube-api-access-2xsgm\") pod \"7276306b-3f0b-49d8-8c54-878c666ac783\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " Oct 31 13:17:50.101369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101346 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-openshift-ca\") pod \"7276306b-3f0b-49d8-8c54-878c666ac783\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " Oct 31 13:17:50.101479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101461 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-xks-ca\") pod \"7276306b-3f0b-49d8-8c54-878c666ac783\" (UID: \"7276306b-3f0b-49d8-8c54-878c666ac783\") " Oct 31 13:17:50.101718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101698 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-envoy\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.101803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101755 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqn2\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-kube-api-access-dlqn2\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.101864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101783 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-openshift-ca" pod "7276306b-3f0b-49d8-8c54-878c666ac783" (UID: "7276306b-3f0b-49d8-8c54-878c666ac783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:50.101864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101793 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-credential-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.101974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101860 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istiod-ca-cert\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.101974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101915 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-data\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.101974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.101958 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-certs\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.102141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102060 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.102201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102147 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-credential-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.103046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102327 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-podinfo\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.103046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102365 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-token\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.103046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102372 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-data\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.103046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102494 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:50.103046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102675 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-socket\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.103046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.102814 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-certs\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.103046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.103001 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istiod-ca-cert\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.105705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.105680 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-envoy\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.105830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.105804 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-podinfo\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.107118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.106734 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7276306b-3f0b-49d8-8c54-878c666ac783-kube-api-access-2xsgm" pod "7276306b-3f0b-49d8-8c54-878c666ac783" (UID: "7276306b-3f0b-49d8-8c54-878c666ac783"). InnerVolumeSpecName "kube-api-access-2xsgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:50.107748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.107716 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-xks-ca" pod "7276306b-3f0b-49d8-8c54-878c666ac783" (UID: "7276306b-3f0b-49d8-8c54-878c666ac783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:17:50.114573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.114542 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqn2\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-kube-api-access-dlqn2\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.115412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.115390 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-token\") pod \"e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.217468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.211572 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsgm\" (UniqueName: \"kubernetes.io/projected/7276306b-3f0b-49d8-8c54-878c666ac783-kube-api-access-2xsgm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:50.217468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.211613 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7276306b-3f0b-49d8-8c54-878c666ac783-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:50.286817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.286781 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:50.580656 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:50.580610 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e181b9_7438_4dcb_9cab_ff8bac239f0e.slice/crio-35e50765f3f271ec1ef234315163e87c92f777c1f03b1bc755b541b2dcc13760 WatchSource:0}: Error finding container 35e50765f3f271ec1ef234315163e87c92f777c1f03b1bc755b541b2dcc13760: Status 404 returned error can't find the container with id 35e50765f3f271ec1ef234315163e87c92f777c1f03b1bc755b541b2dcc13760 Oct 31 13:17:50.581148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.581119 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn"] Oct 31 13:17:50.584741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.584703 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:17:50.584842 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.584789 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:17:50.584842 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.584832 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:17:50.794469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.794420 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" event={"ID":"90e181b9-7438-4dcb-9cab-ff8bac239f0e","Type":"ContainerStarted","Data":"2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0"} Oct 31 13:17:50.794689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.794501 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" event={"ID":"90e181b9-7438-4dcb-9cab-ff8bac239f0e","Type":"ContainerStarted","Data":"35e50765f3f271ec1ef234315163e87c92f777c1f03b1bc755b541b2dcc13760"} Oct 31 13:17:50.796555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.796527 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" Oct 31 13:17:50.796709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.796573 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm" event={"ID":"7276306b-3f0b-49d8-8c54-878c666ac783","Type":"ContainerDied","Data":"7a1e0e6dab005224c695ea921f3d955147d730d5bee68bbdc6eb236e3a657846"} Oct 31 13:17:50.796709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.796672 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1e0e6dab005224c695ea921f3d955147d730d5bee68bbdc6eb236e3a657846" Oct 31 13:17:50.823318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:50.823261 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podStartSLOduration=1.823248103 podStartE2EDuration="1.823248103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:50.817513834 +0000 UTC m=+2764.995231819" watchObservedRunningTime="2026-10-31 13:17:50.823248103 +0000 UTC m=+2765.000966089" Oct 31 13:17:51.287406 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:51.287372 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:51.289073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:51.289045 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:17:51.289231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:51.289102 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 13:17:51.443453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:51.443423 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:17:51.851659 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:17:51.851214 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:17:52.210236 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:17:52.210135 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:17:52.294657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:52.291598 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:17:52.294657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:52.291682 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 13:17:53.029179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.029137 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw"] Oct 31 13:17:53.033593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.033568 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k"] Oct 31 13:17:53.037567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.037548 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942e5f29-7ktlw"] Oct 31 13:17:53.041104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.041082 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35a4a8-26s4k"] Oct 31 13:17:53.299715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.299115 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:17:53.299715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.299179 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 13:17:53.941112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.941064 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcc14a0-2bab-4527-a2d3-3925673b8b45" path="/var/lib/kubelet/pods/0bcc14a0-2bab-4527-a2d3-3925673b8b45/volumes" Oct 31 13:17:53.941682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:53.941655 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87b86da-756e-4713-ac88-2b28764de3b1" path="/var/lib/kubelet/pods/c87b86da-756e-4713-ac88-2b28764de3b1/volumes" Oct 31 13:17:54.301947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:54.301898 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:17:54.302481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:54.301970 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 13:17:55.030338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.030286 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts"] Oct 31 13:17:55.034819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.034729 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76aa74be-wg9ts"] Oct 31 13:17:55.291714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.288718 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:17:55.291714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.288784 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 13:17:55.502449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.502403 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l"] Oct 31 13:17:55.503393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.503366 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7276306b-3f0b-49d8-8c54-878c666ac783" containerName="revoke-keys" Oct 31 13:17:55.503393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.503394 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7276306b-3f0b-49d8-8c54-878c666ac783" containerName="revoke-keys" Oct 31 13:17:55.503608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.503594 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7276306b-3f0b-49d8-8c54-878c666ac783" containerName="revoke-keys" Oct 31 13:17:55.509232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.509208 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.518340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.518315 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:17:55.518460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.518324 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:17:55.528197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.528171 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:17:55.528420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.528405 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:17:55.553116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.553051 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.553431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.553408 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdcc\" (UniqueName: \"kubernetes.io/projected/9d78e531-db74-427e-8bc9-2760cc87c9a8-kube-api-access-pwdcc\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.553547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.553527 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.634390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.634353 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l"] Oct 31 13:17:55.655492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.655452 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdcc\" (UniqueName: \"kubernetes.io/projected/9d78e531-db74-427e-8bc9-2760cc87c9a8-kube-api-access-pwdcc\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.655678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.655572 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.655772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.655702 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.656461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.656434 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.658116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.658090 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.702689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.702656 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdcc\" (UniqueName: \"kubernetes.io/projected/9d78e531-db74-427e-8bc9-2760cc87c9a8-kube-api-access-pwdcc\") pod \"maas-api-revoke-sub-8eb68f83-clk4l\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.837357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.837261 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:55.953093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:55.952920 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9851471-2652-4d93-a001-0288b3905e40" path="/var/lib/kubelet/pods/c9851471-2652-4d93-a001-0288b3905e40/volumes" Oct 31 13:17:56.023457 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:17:56.023395 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:17:56.205303 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:17:56.205256 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d78e531_db74_427e_8bc9_2760cc87c9a8.slice/crio-ca117daf34c3917ee111e584a4f60c6335dcf78deacfb9fc1cc945ad7557c012 WatchSource:0}: Error finding container ca117daf34c3917ee111e584a4f60c6335dcf78deacfb9fc1cc945ad7557c012: Status 404 returned error can't find the container with id ca117daf34c3917ee111e584a4f60c6335dcf78deacfb9fc1cc945ad7557c012 Oct 31 13:17:56.207458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:56.207427 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l"] Oct 31 13:17:56.303657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:56.303076 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:17:56.303657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:56.303140 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 13:17:56.875660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:56.870937 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" event={"ID":"9d78e531-db74-427e-8bc9-2760cc87c9a8","Type":"ContainerStarted","Data":"1320d6377af3c1247f37df3404a38de9d9583dc9e40dad88377e4fa970c05904"} Oct 31 13:17:56.875660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:56.871014 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" event={"ID":"9d78e531-db74-427e-8bc9-2760cc87c9a8","Type":"ContainerStarted","Data":"ca117daf34c3917ee111e584a4f60c6335dcf78deacfb9fc1cc945ad7557c012"} Oct 31 13:17:56.914798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:56.914738 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" podStartSLOduration=1.9147249990000001 podStartE2EDuration="1.914724999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:56.909356394 +0000 UTC m=+2771.087074378" watchObservedRunningTime="2026-10-31 13:17:56.914724999 +0000 UTC m=+2771.092442982" Oct 31 13:17:57.287391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:57.287357 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 31 13:17:57.287551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:57.287422 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 31 13:17:57.442656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:57.442103 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6"] Oct 31 13:17:57.455759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:57.454263 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:17:57.933443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:57.933405 5074 generic.go:356] "Generic (PLEG): container finished" podID="9d78e531-db74-427e-8bc9-2760cc87c9a8" containerID="1320d6377af3c1247f37df3404a38de9d9583dc9e40dad88377e4fa970c05904" exitCode=0 Oct 31 13:17:58.001606 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:58.001566 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" event={"ID":"9d78e531-db74-427e-8bc9-2760cc87c9a8","Type":"ContainerDied","Data":"1320d6377af3c1247f37df3404a38de9d9583dc9e40dad88377e4fa970c05904"} Oct 31 13:17:58.292217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:58.292179 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:58.292709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:58.292669 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:58.293605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:58.293581 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:17:59.199232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.198875 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:17:59.284006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.283974 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-openshift-ca\") pod \"9d78e531-db74-427e-8bc9-2760cc87c9a8\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " Oct 31 13:17:59.284197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.284059 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d78e531-db74-427e-8bc9-2760cc87c9a8-kube-api-access-pwdcc\" (UniqueName: \"kubernetes.io/projected/9d78e531-db74-427e-8bc9-2760cc87c9a8-kube-api-access-pwdcc\") pod \"9d78e531-db74-427e-8bc9-2760cc87c9a8\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " Oct 31 13:17:59.284197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.284096 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-xks-ca\") pod \"9d78e531-db74-427e-8bc9-2760cc87c9a8\" (UID: \"9d78e531-db74-427e-8bc9-2760cc87c9a8\") " Oct 31 13:17:59.284395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.284361 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-openshift-ca" pod "9d78e531-db74-427e-8bc9-2760cc87c9a8" (UID: "9d78e531-db74-427e-8bc9-2760cc87c9a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:17:59.284966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.284937 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:59.286454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.286422 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-xks-ca" pod "9d78e531-db74-427e-8bc9-2760cc87c9a8" (UID: "9d78e531-db74-427e-8bc9-2760cc87c9a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:17:59.286586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.286521 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d78e531-db74-427e-8bc9-2760cc87c9a8-kube-api-access-pwdcc" pod "9d78e531-db74-427e-8bc9-2760cc87c9a8" (UID: "9d78e531-db74-427e-8bc9-2760cc87c9a8"). InnerVolumeSpecName "kube-api-access-pwdcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:17:59.387237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.387196 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdcc\" (UniqueName: \"kubernetes.io/projected/9d78e531-db74-427e-8bc9-2760cc87c9a8-kube-api-access-pwdcc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:59.387237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.387238 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d78e531-db74-427e-8bc9-2760cc87c9a8-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:17:59.786857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.786823 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:17:59.955983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.955943 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" event={"ID":"9d78e531-db74-427e-8bc9-2760cc87c9a8","Type":"ContainerDied","Data":"ca117daf34c3917ee111e584a4f60c6335dcf78deacfb9fc1cc945ad7557c012"} Oct 31 13:17:59.956145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.956008 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca117daf34c3917ee111e584a4f60c6335dcf78deacfb9fc1cc945ad7557c012" Oct 31 13:17:59.956145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:17:59.955961 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l" Oct 31 13:18:02.321337 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:02.321293 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:02.451651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.444400 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" podUID="730af042-bf20-4386-a509-cccd48fb9df1" containerName="payload-processing" containerID="cri-o://5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613" gracePeriod=25 Oct 31 13:18:02.762970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.762858 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:18:02.850924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.850834 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-routing-overlay-volume\") pod \"730af042-bf20-4386-a509-cccd48fb9df1\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " Oct 31 13:18:02.851082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.850947 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-kube-api-access-bsp5k\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-kube-api-access-bsp5k\") pod \"730af042-bf20-4386-a509-cccd48fb9df1\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " Oct 31 13:18:02.851174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.851153 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-provider-credentials\") pod \"730af042-bf20-4386-a509-cccd48fb9df1\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " Oct 31 13:18:02.851237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.851200 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-plugins-config-volume\") pod \"730af042-bf20-4386-a509-cccd48fb9df1\" (UID: \"730af042-bf20-4386-a509-cccd48fb9df1\") " Oct 31 13:18:02.851763 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.851730 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-plugins-config-volume" pod "730af042-bf20-4386-a509-cccd48fb9df1" (UID: "730af042-bf20-4386-a509-cccd48fb9df1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:02.851929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.851777 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-routing-overlay-volume" pod "730af042-bf20-4386-a509-cccd48fb9df1" (UID: "730af042-bf20-4386-a509-cccd48fb9df1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:02.851979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.851954 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:02.852019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.851975 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730af042-bf20-4386-a509-cccd48fb9df1-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:02.853422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.853389 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-kube-api-access-bsp5k" pod "730af042-bf20-4386-a509-cccd48fb9df1" (UID: "730af042-bf20-4386-a509-cccd48fb9df1"). InnerVolumeSpecName "kube-api-access-bsp5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:02.853729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.853707 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-provider-credentials" pod "730af042-bf20-4386-a509-cccd48fb9df1" (UID: "730af042-bf20-4386-a509-cccd48fb9df1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:02.953297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.953261 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsp5k\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-kube-api-access-bsp5k\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:02.953297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.953296 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/730af042-bf20-4386-a509-cccd48fb9df1-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:02.988815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.988780 5074 generic.go:356] "Generic (PLEG): container finished" podID="730af042-bf20-4386-a509-cccd48fb9df1" containerID="5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613" exitCode=0 Oct 31 13:18:02.988986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.988820 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" Oct 31 13:18:02.988986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.988866 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" event={"ID":"730af042-bf20-4386-a509-cccd48fb9df1","Type":"ContainerDied","Data":"5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613"} Oct 31 13:18:02.988986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.988925 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6" event={"ID":"730af042-bf20-4386-a509-cccd48fb9df1","Type":"ContainerDied","Data":"90a2d6cf07bab3534c4665df9c2821c3518530fe77c88cc2cc7a9c1c17f4ec17"} Oct 31 13:18:02.988986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:02.988953 5074 scope.go:122] "RemoveContainer" containerID="5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613" Oct 31 13:18:03.001298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:03.001277 5074 scope.go:122] "RemoveContainer" containerID="5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613" Oct 31 13:18:03.001597 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:03.001575 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613\": container with ID starting with 5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613 not found: ID does not exist" containerID="5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613" Oct 31 13:18:03.001668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:03.001606 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613"} err="failed to get container status \"5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613\": rpc error: code = NotFound desc = could not find container \"5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613\": container with ID starting with 5e1a80638de7bbfe7c123749c88ea4a1874c5eb65f84b9de341b5cd62ae06613 not found: ID does not exist" Oct 31 13:18:03.016391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:03.016364 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6"] Oct 31 13:18:03.020565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:03.020543 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0c1fbb/payload-processing-external-model-e2e-extmod-b-0c1fbb-7d4bvzbc6"] Oct 31 13:18:03.941553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:03.941516 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730af042-bf20-4386-a509-cccd48fb9df1" path="/var/lib/kubelet/pods/730af042-bf20-4386-a509-cccd48fb9df1/volumes" Oct 31 13:18:06.024009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.023971 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn"] Oct 31 13:18:06.024747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.024729 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9d78e531-db74-427e-8bc9-2760cc87c9a8" containerName="revoke-keys" Oct 31 13:18:06.024747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.024747 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d78e531-db74-427e-8bc9-2760cc87c9a8" containerName="revoke-keys" Oct 31 13:18:06.024868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.024772 5074 cpu_manager.go:414] "containerMap: removing container" podUID="730af042-bf20-4386-a509-cccd48fb9df1" containerName="payload-processing" Oct 31 13:18:06.024868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.024777 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730af042-bf20-4386-a509-cccd48fb9df1" containerName="payload-processing" Oct 31 13:18:06.024933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.024901 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d78e531-db74-427e-8bc9-2760cc87c9a8" containerName="revoke-keys" Oct 31 13:18:06.024933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.024913 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="730af042-bf20-4386-a509-cccd48fb9df1" containerName="payload-processing" Oct 31 13:18:06.030177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.030153 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.035986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.035958 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:18:06.036540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.036500 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:18:06.037444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.037421 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn"] Oct 31 13:18:06.039611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.039590 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:18:06.039747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.039683 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:18:06.157165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.157130 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803445e-6131-4e74-a493-d74dda90b945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.157347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.157248 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803445e-6131-4e74-a493-d74dda90b945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.157347 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.157322 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktr4q\" (UniqueName: \"kubernetes.io/projected/4803445e-6131-4e74-a493-d74dda90b945-kube-api-access-ktr4q\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.258465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.258426 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803445e-6131-4e74-a493-d74dda90b945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.258685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.258533 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktr4q\" (UniqueName: \"kubernetes.io/projected/4803445e-6131-4e74-a493-d74dda90b945-kube-api-access-ktr4q\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.258822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.258798 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803445e-6131-4e74-a493-d74dda90b945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.259251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.259224 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803445e-6131-4e74-a493-d74dda90b945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.261348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.261323 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803445e-6131-4e74-a493-d74dda90b945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.281699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.281608 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktr4q\" (UniqueName: \"kubernetes.io/projected/4803445e-6131-4e74-a493-d74dda90b945-kube-api-access-ktr4q\") pod \"maas-api-revoke-sub-efc20555-bqxsn\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.374464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.374426 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:06.680197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:06.680122 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn"] Oct 31 13:18:06.684483 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:18:06.684450 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4803445e_6131_4e74_a493_d74dda90b945.slice/crio-81a33b8a1d9383c88de40ac9eef2322733bfe03e4940b204338f9f1de57351f9 WatchSource:0}: Error finding container 81a33b8a1d9383c88de40ac9eef2322733bfe03e4940b204338f9f1de57351f9: Status 404 returned error can't find the container with id 81a33b8a1d9383c88de40ac9eef2322733bfe03e4940b204338f9f1de57351f9 Oct 31 13:18:06.721330 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:06.721304 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:07.058605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:07.058556 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" event={"ID":"4803445e-6131-4e74-a493-d74dda90b945","Type":"ContainerStarted","Data":"81a33b8a1d9383c88de40ac9eef2322733bfe03e4940b204338f9f1de57351f9"} Oct 31 13:18:08.108904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:08.108860 5074 generic.go:356] "Generic (PLEG): container finished" podID="4803445e-6131-4e74-a493-d74dda90b945" containerID="0676c7b06938c8e6211059d8c1ea79c45b0c6110c25671659b0b7144f9b3c0da" exitCode=0 Oct 31 13:18:08.109504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:08.109139 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" event={"ID":"4803445e-6131-4e74-a493-d74dda90b945","Type":"ContainerDied","Data":"0676c7b06938c8e6211059d8c1ea79c45b0c6110c25671659b0b7144f9b3c0da"} Oct 31 13:18:09.333205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.333178 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:09.524144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.524105 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4803445e-6131-4e74-a493-d74dda90b945-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803445e-6131-4e74-a493-d74dda90b945-maas-api-openshift-ca\") pod \"4803445e-6131-4e74-a493-d74dda90b945\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " Oct 31 13:18:09.524343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.524258 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4803445e-6131-4e74-a493-d74dda90b945-kube-api-access-ktr4q\" (UniqueName: \"kubernetes.io/projected/4803445e-6131-4e74-a493-d74dda90b945-kube-api-access-ktr4q\") pod \"4803445e-6131-4e74-a493-d74dda90b945\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " Oct 31 13:18:09.524343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.524332 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4803445e-6131-4e74-a493-d74dda90b945-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803445e-6131-4e74-a493-d74dda90b945-maas-api-xks-ca\") pod \"4803445e-6131-4e74-a493-d74dda90b945\" (UID: \"4803445e-6131-4e74-a493-d74dda90b945\") " Oct 31 13:18:09.524529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.524504 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4803445e-6131-4e74-a493-d74dda90b945-maas-api-openshift-ca" pod "4803445e-6131-4e74-a493-d74dda90b945" (UID: "4803445e-6131-4e74-a493-d74dda90b945"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:09.525017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.524995 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803445e-6131-4e74-a493-d74dda90b945-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:09.526438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.526416 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4803445e-6131-4e74-a493-d74dda90b945-kube-api-access-ktr4q" pod "4803445e-6131-4e74-a493-d74dda90b945" (UID: "4803445e-6131-4e74-a493-d74dda90b945"). InnerVolumeSpecName "kube-api-access-ktr4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:09.526620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.526604 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4803445e-6131-4e74-a493-d74dda90b945-maas-api-xks-ca" pod "4803445e-6131-4e74-a493-d74dda90b945" (UID: "4803445e-6131-4e74-a493-d74dda90b945"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:18:09.626318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.626281 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktr4q\" (UniqueName: \"kubernetes.io/projected/4803445e-6131-4e74-a493-d74dda90b945-kube-api-access-ktr4q\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:09.626485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:09.626320 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803445e-6131-4e74-a493-d74dda90b945-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:10.129561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:10.129532 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" Oct 31 13:18:10.129561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:10.129551 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn" event={"ID":"4803445e-6131-4e74-a493-d74dda90b945","Type":"ContainerDied","Data":"81a33b8a1d9383c88de40ac9eef2322733bfe03e4940b204338f9f1de57351f9"} Oct 31 13:18:10.129819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:10.129603 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81a33b8a1d9383c88de40ac9eef2322733bfe03e4940b204338f9f1de57351f9" Oct 31 13:18:12.419151 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:12.419113 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:18.691475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.691431 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5"] Oct 31 13:18:18.692592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.692570 5074 cpu_manager.go:414] "containerMap: removing container" podUID="4803445e-6131-4e74-a493-d74dda90b945" containerName="revoke-keys" Oct 31 13:18:18.692703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.692596 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4803445e-6131-4e74-a493-d74dda90b945" containerName="revoke-keys" Oct 31 13:18:18.692849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.692835 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4803445e-6131-4e74-a493-d74dda90b945" containerName="revoke-keys" Oct 31 13:18:18.697548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.697527 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.705669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.705625 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5"] Oct 31 13:18:18.706612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.706587 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:18:18.706777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.706704 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:18:18.706777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.706737 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:18:18.708225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.708191 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:18:18.778727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.777734 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.778727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.777893 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.778727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.777969 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpfg\" (UniqueName: \"kubernetes.io/projected/94e94e2e-bdbb-423f-99e8-5fa920cfad10-kube-api-access-8vpfg\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.880368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.880326 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.880571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.880424 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpfg\" (UniqueName: \"kubernetes.io/projected/94e94e2e-bdbb-423f-99e8-5fa920cfad10-kube-api-access-8vpfg\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.880717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.880696 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.881663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.881541 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.884860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.884832 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:18.906972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:18.906939 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpfg\" (UniqueName: \"kubernetes.io/projected/94e94e2e-bdbb-423f-99e8-5fa920cfad10-kube-api-access-8vpfg\") pod \"maas-api-revoke-sub-c8fd992d-6brd5\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:19.022494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:19.022452 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:19.350956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:19.350891 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5"] Oct 31 13:18:19.354202 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:18:19.354166 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e94e2e_bdbb_423f_99e8_5fa920cfad10.slice/crio-f4336d34249d749c2fc3959865c8f76e31661a7c04c84b3884e48fb1abc7a37e WatchSource:0}: Error finding container f4336d34249d749c2fc3959865c8f76e31661a7c04c84b3884e48fb1abc7a37e: Status 404 returned error can't find the container with id f4336d34249d749c2fc3959865c8f76e31661a7c04c84b3884e48fb1abc7a37e Oct 31 13:18:19.425228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:19.425196 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" event={"ID":"94e94e2e-bdbb-423f-99e8-5fa920cfad10","Type":"ContainerStarted","Data":"f4336d34249d749c2fc3959865c8f76e31661a7c04c84b3884e48fb1abc7a37e"} Oct 31 13:18:19.943397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:19.943362 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:18:20.438801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:20.438761 5074 generic.go:356] "Generic (PLEG): container finished" podID="94e94e2e-bdbb-423f-99e8-5fa920cfad10" containerID="17cf5b0f3a014e8d72b299c8d8804cc234336ef9829c72d05178648e0a672b9c" exitCode=0 Oct 31 13:18:20.439005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:20.438916 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" event={"ID":"94e94e2e-bdbb-423f-99e8-5fa920cfad10","Type":"ContainerDied","Data":"17cf5b0f3a014e8d72b299c8d8804cc234336ef9829c72d05178648e0a672b9c"} Oct 31 13:18:21.741323 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:21.741236 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:21.778532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.778507 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:21.817725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.817690 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-xks-ca\") pod \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " Oct 31 13:18:21.818028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.818008 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-openshift-ca\") pod \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " Oct 31 13:18:21.818149 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.818124 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e94e2e-bdbb-423f-99e8-5fa920cfad10-kube-api-access-8vpfg\" (UniqueName: \"kubernetes.io/projected/94e94e2e-bdbb-423f-99e8-5fa920cfad10-kube-api-access-8vpfg\") pod \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\" (UID: \"94e94e2e-bdbb-423f-99e8-5fa920cfad10\") " Oct 31 13:18:21.818444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.818416 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-openshift-ca" pod "94e94e2e-bdbb-423f-99e8-5fa920cfad10" (UID: "94e94e2e-bdbb-423f-99e8-5fa920cfad10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:21.819008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.818985 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:21.820162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.820130 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-xks-ca" pod "94e94e2e-bdbb-423f-99e8-5fa920cfad10" (UID: "94e94e2e-bdbb-423f-99e8-5fa920cfad10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:18:21.821050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.820766 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e94e2e-bdbb-423f-99e8-5fa920cfad10-kube-api-access-8vpfg" pod "94e94e2e-bdbb-423f-99e8-5fa920cfad10" (UID: "94e94e2e-bdbb-423f-99e8-5fa920cfad10"). InnerVolumeSpecName "kube-api-access-8vpfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:21.920403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.920360 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpfg\" (UniqueName: \"kubernetes.io/projected/94e94e2e-bdbb-423f-99e8-5fa920cfad10-kube-api-access-8vpfg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:21.920403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:21.920397 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e94e2e-bdbb-423f-99e8-5fa920cfad10-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:22.437075 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:22.437037 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:22.458580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:22.458548 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" event={"ID":"94e94e2e-bdbb-423f-99e8-5fa920cfad10","Type":"ContainerDied","Data":"f4336d34249d749c2fc3959865c8f76e31661a7c04c84b3884e48fb1abc7a37e"} Oct 31 13:18:22.458580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:22.458574 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5" Oct 31 13:18:22.458794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:22.458603 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4336d34249d749c2fc3959865c8f76e31661a7c04c84b3884e48fb1abc7a37e" Oct 31 13:18:23.723178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.723139 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9"] Oct 31 13:18:23.724492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.724465 5074 cpu_manager.go:414] "containerMap: removing container" podUID="94e94e2e-bdbb-423f-99e8-5fa920cfad10" containerName="revoke-keys" Oct 31 13:18:23.724492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.724491 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e94e2e-bdbb-423f-99e8-5fa920cfad10" containerName="revoke-keys" Oct 31 13:18:23.724759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.724742 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e94e2e-bdbb-423f-99e8-5fa920cfad10" containerName="revoke-keys" Oct 31 13:18:23.728732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.728708 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.733147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.733118 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:18:23.734089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.734011 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:18:23.734298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.734248 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:18:23.734405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.734319 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:18:23.736055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.736032 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9"] Oct 31 13:18:23.856597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.856557 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4ft\" (UniqueName: \"kubernetes.io/projected/844d1780-a71f-4847-92c3-7fac1ce625c3-kube-api-access-5j4ft\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.856796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.856751 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.856862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.856797 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.958608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.958574 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.958805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.958620 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.958922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.958902 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4ft\" (UniqueName: \"kubernetes.io/projected/844d1780-a71f-4847-92c3-7fac1ce625c3-kube-api-access-5j4ft\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.959351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.959331 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.961191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.961170 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:23.971511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:23.971486 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4ft\" (UniqueName: \"kubernetes.io/projected/844d1780-a71f-4847-92c3-7fac1ce625c3-kube-api-access-5j4ft\") pod \"maas-api-revoke-sub-3d2267da-xmhn9\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:24.051019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:24.050950 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:24.342689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:24.342563 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9"] Oct 31 13:18:24.499040 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:24.499002 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" event={"ID":"844d1780-a71f-4847-92c3-7fac1ce625c3","Type":"ContainerStarted","Data":"a8c7d265e736448b22e4589d1783b41f7ba7a464296ebf723bd3a3c2f3534cdf"} Oct 31 13:18:24.538386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:24.538350 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm"] Oct 31 13:18:24.538666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:24.538597 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" containerID="cri-o://8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc" gracePeriod=30 Oct 31 13:18:25.508970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:25.508927 5074 generic.go:356] "Generic (PLEG): container finished" podID="844d1780-a71f-4847-92c3-7fac1ce625c3" containerID="fdff75e16c9bec78182094babaa0a45ebacc4f9ba13aae0e5b6b798d6e715ccb" exitCode=0 Oct 31 13:18:25.509375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:25.509242 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" event={"ID":"844d1780-a71f-4847-92c3-7fac1ce625c3","Type":"ContainerDied","Data":"fdff75e16c9bec78182094babaa0a45ebacc4f9ba13aae0e5b6b798d6e715ccb"} Oct 31 13:18:25.984041 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:25.984000 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:18:26.799015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.798981 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:26.926600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.926566 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844d1780-a71f-4847-92c3-7fac1ce625c3-kube-api-access-5j4ft\" (UniqueName: \"kubernetes.io/projected/844d1780-a71f-4847-92c3-7fac1ce625c3-kube-api-access-5j4ft\") pod \"844d1780-a71f-4847-92c3-7fac1ce625c3\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " Oct 31 13:18:26.926819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.926629 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-openshift-ca\") pod \"844d1780-a71f-4847-92c3-7fac1ce625c3\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " Oct 31 13:18:26.926819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.926690 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-xks-ca\") pod \"844d1780-a71f-4847-92c3-7fac1ce625c3\" (UID: \"844d1780-a71f-4847-92c3-7fac1ce625c3\") " Oct 31 13:18:26.927083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.927055 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-openshift-ca" pod "844d1780-a71f-4847-92c3-7fac1ce625c3" (UID: "844d1780-a71f-4847-92c3-7fac1ce625c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:26.927519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.927499 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:26.929071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.929038 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-xks-ca" pod "844d1780-a71f-4847-92c3-7fac1ce625c3" (UID: "844d1780-a71f-4847-92c3-7fac1ce625c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:18:26.929358 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:26.929331 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844d1780-a71f-4847-92c3-7fac1ce625c3-kube-api-access-5j4ft" pod "844d1780-a71f-4847-92c3-7fac1ce625c3" (UID: "844d1780-a71f-4847-92c3-7fac1ce625c3"). InnerVolumeSpecName "kube-api-access-5j4ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:27.029595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:27.029509 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4ft\" (UniqueName: \"kubernetes.io/projected/844d1780-a71f-4847-92c3-7fac1ce625c3-kube-api-access-5j4ft\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:27.029595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:27.029548 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844d1780-a71f-4847-92c3-7fac1ce625c3-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:27.545566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:27.544489 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" Oct 31 13:18:27.545566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:27.544514 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9" event={"ID":"844d1780-a71f-4847-92c3-7fac1ce625c3","Type":"ContainerDied","Data":"a8c7d265e736448b22e4589d1783b41f7ba7a464296ebf723bd3a3c2f3534cdf"} Oct 31 13:18:27.545566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:27.544566 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8c7d265e736448b22e4589d1783b41f7ba7a464296ebf723bd3a3c2f3534cdf" Oct 31 13:18:29.860772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:29.860722 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz"] Oct 31 13:18:29.906093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:29.904194 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:18:30.114870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.114765 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-certs\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.115167 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.115140 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-istio-token\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-istio-token\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.115377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.115355 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-data\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.115597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.115577 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-kube-api-access-2v6wn\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-kube-api-access-2v6wn\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.115967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.115945 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cbde43-0ea8-4353-aec0-1520301c889f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbde43-0ea8-4353-aec0-1520301c889f-istiod-ca-cert\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.116099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.116085 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-socket\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.116225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.116212 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85cbde43-0ea8-4353-aec0-1520301c889f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbde43-0ea8-4353-aec0-1520301c889f-istio-podinfo\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.116340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.116325 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-envoy\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.116450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.116437 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-credential-socket\") pod \"85cbde43-0ea8-4353-aec0-1520301c889f\" (UID: \"85cbde43-0ea8-4353-aec0-1520301c889f\") " Oct 31 13:18:30.126259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.126228 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-certs" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:30.127073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.127051 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-credential-socket" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:30.133393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.129869 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-data" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:30.160220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.160179 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-envoy" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:30.162269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.161035 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-socket" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:30.163165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.163140 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cbde43-0ea8-4353-aec0-1520301c889f-istiod-ca-cert" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:30.203657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.184765 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85cbde43-0ea8-4353-aec0-1520301c889f-istio-podinfo" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:18:30.203657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.185127 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-kube-api-access-2v6wn" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "kube-api-access-2v6wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:30.203657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.185856 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-istio-token" pod "85cbde43-0ea8-4353-aec0-1520301c889f" (UID: "85cbde43-0ea8-4353-aec0-1520301c889f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256046 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256091 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v6wn\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-kube-api-access-2v6wn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256110 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbde43-0ea8-4353-aec0-1520301c889f-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256125 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256139 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbde43-0ea8-4353-aec0-1520301c889f-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256154 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256168 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256183 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbde43-0ea8-4353-aec0-1520301c889f-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.258405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.256196 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbde43-0ea8-4353-aec0-1520301c889f-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:30.572920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.572879 5074 generic.go:356] "Generic (PLEG): container finished" podID="85cbde43-0ea8-4353-aec0-1520301c889f" containerID="8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc" exitCode=0 Oct 31 13:18:30.573096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.572941 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" Oct 31 13:18:30.573096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.572965 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" event={"ID":"85cbde43-0ea8-4353-aec0-1520301c889f","Type":"ContainerDied","Data":"8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc"} Oct 31 13:18:30.573096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.573035 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm" event={"ID":"85cbde43-0ea8-4353-aec0-1520301c889f","Type":"ContainerDied","Data":"f2ae22948fb9ef9e2ba197f31cd4933743539ba34574d550f37680571988a12d"} Oct 31 13:18:30.573096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.573064 5074 scope.go:122] "RemoveContainer" containerID="8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc" Oct 31 13:18:30.585960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.585942 5074 scope.go:122] "RemoveContainer" containerID="8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc" Oct 31 13:18:30.586276 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:30.586258 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc\": container with ID starting with 8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc not found: ID does not exist" containerID="8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc" Oct 31 13:18:30.586353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.586282 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc"} err="failed to get container status \"8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc\": rpc error: code = NotFound desc = could not find container \"8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc\": container with ID starting with 8c44929286b8b30d31dac4589f4bb1534f79ca9e5487196fc0047ae7caeda8cc not found: ID does not exist" Oct 31 13:18:30.603247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.603217 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm"] Oct 31 13:18:30.610826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:30.610795 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-f9msm"] Oct 31 13:18:31.941258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:31.941223 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" path="/var/lib/kubelet/pods/85cbde43-0ea8-4353-aec0-1520301c889f/volumes" Oct 31 13:18:32.536168 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:32.536129 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:34.862051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:34.862000 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" podUID="bd08ad70-09bd-4903-9b55-8145aab7800e" containerName="payload-processing" containerID="cri-o://0264697b7fb4c4d79cfdb065d5550f642995a3eee1451fd45d042a077d7d0ad3" gracePeriod=25 Oct 31 13:18:35.761850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.761810 5074 generic.go:356] "Generic (PLEG): container finished" podID="bd08ad70-09bd-4903-9b55-8145aab7800e" containerID="0264697b7fb4c4d79cfdb065d5550f642995a3eee1451fd45d042a077d7d0ad3" exitCode=0 Oct 31 13:18:35.762026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.761864 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" event={"ID":"bd08ad70-09bd-4903-9b55-8145aab7800e","Type":"ContainerDied","Data":"0264697b7fb4c4d79cfdb065d5550f642995a3eee1451fd45d042a077d7d0ad3"} Oct 31 13:18:35.773547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.773207 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:18:35.855317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.855285 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-routing-overlay-volume\") pod \"bd08ad70-09bd-4903-9b55-8145aab7800e\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " Oct 31 13:18:35.855508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.855497 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-kube-api-access-t28jc\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-kube-api-access-t28jc\") pod \"bd08ad70-09bd-4903-9b55-8145aab7800e\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " Oct 31 13:18:35.855554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.855520 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-plugins-config-volume\") pod \"bd08ad70-09bd-4903-9b55-8145aab7800e\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " Oct 31 13:18:35.855616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.855592 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-provider-credentials\") pod \"bd08ad70-09bd-4903-9b55-8145aab7800e\" (UID: \"bd08ad70-09bd-4903-9b55-8145aab7800e\") " Oct 31 13:18:35.855720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.855691 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-routing-overlay-volume" pod "bd08ad70-09bd-4903-9b55-8145aab7800e" (UID: "bd08ad70-09bd-4903-9b55-8145aab7800e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:35.855908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.855885 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-plugins-config-volume" pod "bd08ad70-09bd-4903-9b55-8145aab7800e" (UID: "bd08ad70-09bd-4903-9b55-8145aab7800e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:35.856214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.856199 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:35.856265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.856218 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd08ad70-09bd-4903-9b55-8145aab7800e-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:35.857563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.857542 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-provider-credentials" pod "bd08ad70-09bd-4903-9b55-8145aab7800e" (UID: "bd08ad70-09bd-4903-9b55-8145aab7800e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:35.857674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.857569 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-kube-api-access-t28jc" pod "bd08ad70-09bd-4903-9b55-8145aab7800e" (UID: "bd08ad70-09bd-4903-9b55-8145aab7800e"). InnerVolumeSpecName "kube-api-access-t28jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:35.963718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.958002 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t28jc\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-kube-api-access-t28jc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:35.963718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:35.958064 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd08ad70-09bd-4903-9b55-8145aab7800e-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:36.432051 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:36.432007 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:36.432238 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:36.432162 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:36.738314 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:36.738278 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:36.772605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:36.772420 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" Oct 31 13:18:36.772605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:36.772456 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz" event={"ID":"bd08ad70-09bd-4903-9b55-8145aab7800e","Type":"ContainerDied","Data":"87c7af2883886b736d59c818f1978645931f2b304ff92a7645ee6b076058e75f"} Oct 31 13:18:36.772605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:36.772539 5074 scope.go:122] "RemoveContainer" containerID="0264697b7fb4c4d79cfdb065d5550f642995a3eee1451fd45d042a077d7d0ad3" Oct 31 13:18:36.813713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:36.813678 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz"] Oct 31 13:18:36.828529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:36.828494 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-7f79vbfdz"] Oct 31 13:18:38.018315 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:38.018274 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd08ad70-09bd-4903-9b55-8145aab7800e" path="/var/lib/kubelet/pods/bd08ad70-09bd-4903-9b55-8145aab7800e/volumes" Oct 31 13:18:42.643528 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:42.643490 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2abba_74f9_4487_9fc7_f9b0bca247e8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:18:50.992912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:50.992878 5074 scope.go:122] "RemoveContainer" containerID="3d92350de9d6e688a9b0bb6a4870d0ed9aa99937cd363aeb02509839390927b8" Oct 31 13:18:51.004912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:51.004890 5074 scope.go:122] "RemoveContainer" containerID="d5371fe9659b219bbd9aa72db434681c77fe63555044be137ca6c3eebc470177" Oct 31 13:18:51.016408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:51.016394 5074 scope.go:122] "RemoveContainer" containerID="ec64e76eac77395fdac4bec94db747b869720290196a81473beaa280cdca9328" Oct 31 13:18:51.028336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:51.028314 5074 scope.go:122] "RemoveContainer" containerID="667642ac50fd1bc3785d587d04ff7904b52b752b0b45cb9ef26be6c153cbb473" Oct 31 13:18:51.038679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:51.038654 5074 scope.go:122] "RemoveContainer" containerID="e61ba8a1c9e21f1ef78304ab830991fdb081b596b886148ac89a5ca7c8e65780" Oct 31 13:18:51.049456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:51.049438 5074 scope.go:122] "RemoveContainer" containerID="dfc83079c05362b7e6d6578b1414e0ee37f105494b41694227632cba599c9857" Oct 31 13:18:51.058556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:51.058537 5074 scope.go:122] "RemoveContainer" containerID="3d864e068378198937417485491dcbe8c9032484e3e0292e44308376fb1b8a1f" Oct 31 13:18:52.198748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:52.198710 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct"] Oct 31 13:18:52.199239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:52.198909 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" containerID="cri-o://2e0d19165e6f000c5f06712d21a5ecf07a3ee36eb2039cfc7d06e6ee6f094af2" gracePeriod=30 Oct 31 13:18:54.315378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.315342 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2"] Oct 31 13:18:54.316228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316203 5074 cpu_manager.go:414] "containerMap: removing container" podUID="844d1780-a71f-4847-92c3-7fac1ce625c3" containerName="revoke-keys" Oct 31 13:18:54.316228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316226 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844d1780-a71f-4847-92c3-7fac1ce625c3" containerName="revoke-keys" Oct 31 13:18:54.316409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316251 5074 cpu_manager.go:414] "containerMap: removing container" podUID="bd08ad70-09bd-4903-9b55-8145aab7800e" containerName="payload-processing" Oct 31 13:18:54.316409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316258 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd08ad70-09bd-4903-9b55-8145aab7800e" containerName="payload-processing" Oct 31 13:18:54.316409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316281 5074 cpu_manager.go:414] "containerMap: removing container" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" Oct 31 13:18:54.316409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316287 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" Oct 31 13:18:54.316613 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316494 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="844d1780-a71f-4847-92c3-7fac1ce625c3" containerName="revoke-keys" Oct 31 13:18:54.316613 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316514 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd08ad70-09bd-4903-9b55-8145aab7800e" containerName="payload-processing" Oct 31 13:18:54.316613 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.316528 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cbde43-0ea8-4353-aec0-1520301c889f" containerName="istio-proxy" Oct 31 13:18:54.323423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.323386 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.338344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.338316 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2"] Oct 31 13:18:54.426860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.426828 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-842ac9-openshift-default-dockercfg-tdrsf\"" Oct 31 13:18:54.464213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464179 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2814f-e408-4a17-9e98-39c7227ee416-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464224 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464292 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464349 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjbc\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-kube-api-access-xwjbc\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464395 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464436 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464465 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464595 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.464768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.464705 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.565917 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566061 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjbc\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-kube-api-access-xwjbc\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566102 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566148 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566190 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566296 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566379 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566573 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2814f-e408-4a17-9e98-39c7227ee416-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566616 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.566889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.566801 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.567028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.567004 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-certs\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.567403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.567283 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-data\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.567573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.567550 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2814f-e408-4a17-9e98-39c7227ee416-istiod-ca-cert\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.567715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.567550 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-credential-socket\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.569382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.569357 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-podinfo\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.569494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.569476 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-envoy\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.582321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.582294 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjbc\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-kube-api-access-xwjbc\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.585350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.585325 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-token\") pod \"e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.645305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.645281 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:54.954148 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:18:54.954115 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a2814f_e408_4a17_9e98_39c7227ee416.slice/crio-ddd57720adf98b12bceab3962d81305a78e210205db0bcaf2ab62a38363292e7 WatchSource:0}: Error finding container ddd57720adf98b12bceab3962d81305a78e210205db0bcaf2ab62a38363292e7: Status 404 returned error can't find the container with id ddd57720adf98b12bceab3962d81305a78e210205db0bcaf2ab62a38363292e7 Oct 31 13:18:54.954148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.954141 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2"] Oct 31 13:18:54.957412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.957374 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:18:54.957488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.957472 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:18:54.957538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.957517 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:18:54.975295 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:54.975258 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" event={"ID":"a7a2814f-e408-4a17-9e98-39c7227ee416","Type":"ContainerStarted","Data":"ddd57720adf98b12bceab3962d81305a78e210205db0bcaf2ab62a38363292e7"} Oct 31 13:18:56.368007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:56.366443 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" event={"ID":"a7a2814f-e408-4a17-9e98-39c7227ee416","Type":"ContainerStarted","Data":"2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950"} Oct 31 13:18:56.395941 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:18:56.395889 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:18:56.531656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:56.512550 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podStartSLOduration=2.51252635 podStartE2EDuration="2.51252635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:56.482020002 +0000 UTC m=+2830.659737987" watchObservedRunningTime="2026-10-31 13:18:56.51252635 +0000 UTC m=+2830.690244335" Oct 31 13:18:56.659713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:56.652754 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:18:56.659713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:56.652911 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 13:18:56.659713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:56.652957 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 13:18:57.412483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.412437 5074 generic.go:356] "Generic (PLEG): container finished" podID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerID="2e0d19165e6f000c5f06712d21a5ecf07a3ee36eb2039cfc7d06e6ee6f094af2" exitCode=0 Oct 31 13:18:57.416654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.415165 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" event={"ID":"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96","Type":"ContainerDied","Data":"2e0d19165e6f000c5f06712d21a5ecf07a3ee36eb2039cfc7d06e6ee6f094af2"} Oct 31 13:18:57.554176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.554151 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 13:18:57.657655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.652131 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 13:18:57.657655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.652201 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 13:18:57.724916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.717159 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-token\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-token\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.724916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.717253 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-socket\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.724916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.717289 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-certs\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.724916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.717389 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istiod-ca-cert\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.724916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.717437 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-data\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.724916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.717495 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-credential-socket\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.729181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.728217 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-credential-socket" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:57.729181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.728398 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-kube-api-access-n2qc2\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-kube-api-access-n2qc2\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.729181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.728531 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-envoy\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.729181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.728558 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-socket" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:57.729181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.728607 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-podinfo\") pod \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\" (UID: \"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96\") " Oct 31 13:18:57.729181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.728897 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istiod-ca-cert" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:18:57.729574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.729467 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.729574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.729493 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.729574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.729510 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.729755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.729602 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-certs" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:57.731283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.731142 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-data" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:57.732154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.732121 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-token" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:57.736327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.736291 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-envoy" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:18:57.738681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.738620 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-kube-api-access-n2qc2" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "kube-api-access-n2qc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:18:57.739250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.739224 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-podinfo" pod "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" (UID: "25cd1bd6-c4c3-4c73-9110-4f0b2f604c96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:18:57.831236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.831111 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.831236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.831178 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qc2\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-kube-api-access-n2qc2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.831236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.831194 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.831236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.831209 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.831236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.831221 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:57.831236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:57.831233 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:18:58.424216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:58.424176 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" Oct 31 13:18:58.424799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:58.424766 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct" event={"ID":"25cd1bd6-c4c3-4c73-9110-4f0b2f604c96","Type":"ContainerDied","Data":"21c5d568339a8196e58d1efd824a921660e2d830068143756e39d9988e76808a"} Oct 31 13:18:58.424911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:58.424844 5074 scope.go:122] "RemoveContainer" containerID="2e0d19165e6f000c5f06712d21a5ecf07a3ee36eb2039cfc7d06e6ee6f094af2" Oct 31 13:18:58.448700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:58.448658 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct"] Oct 31 13:18:58.454241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:58.454220 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-7nqct"] Oct 31 13:18:58.646089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:58.646054 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 13:18:58.646261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:58.646107 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 13:18:59.646268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:59.646228 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 13:18:59.646761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:59.646285 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 13:18:59.942697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:18:59.942599 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" path="/var/lib/kubelet/pods/25cd1bd6-c4c3-4c73-9110-4f0b2f604c96/volumes" Oct 31 13:19:00.648656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:00.647987 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 13:19:00.648656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:00.648063 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 13:19:01.645983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:01.645933 5074 patch_prober.go:29] interesting pod/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 13:19:01.646250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:01.646008 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 13:19:02.651016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:02.650985 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:19:02.651514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:02.651496 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:19:02.652285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:02.652260 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:19:03.604783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:03.604748 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:19:05.046659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.036690 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g"] Oct 31 13:19:05.046659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.037854 5074 cpu_manager.go:414] "containerMap: removing container" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" Oct 31 13:19:05.046659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.037879 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" Oct 31 13:19:05.046659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.038129 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cd1bd6-c4c3-4c73-9110-4f0b2f604c96" containerName="istio-proxy" Oct 31 13:19:05.046659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.044545 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.055020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.054987 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:19:05.055422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.055404 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:19:05.067656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.066378 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:19:05.076667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.072856 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:19:05.124388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.124350 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g"] Oct 31 13:19:05.199687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.181502 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.199687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.181681 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttj5k\" (UniqueName: \"kubernetes.io/projected/ba157151-94be-49d2-a2ba-7a0e84091f3a-kube-api-access-ttj5k\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.199687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.181863 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.435141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.435051 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.435889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.435842 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.436206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.436183 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttj5k\" (UniqueName: \"kubernetes.io/projected/ba157151-94be-49d2-a2ba-7a0e84091f3a-kube-api-access-ttj5k\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.446093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.446061 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.448391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.448328 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.459121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.459093 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttj5k\" (UniqueName: \"kubernetes.io/projected/ba157151-94be-49d2-a2ba-7a0e84091f3a-kube-api-access-ttj5k\") pod \"maas-api-revoke-sub-161ad445-78n7g\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:05.722661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:05.722613 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:06.013786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:06.013752 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g"] Oct 31 13:19:06.014173 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:19:06.014144 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba157151_94be_49d2_a2ba_7a0e84091f3a.slice/crio-0c5a6c41c3f495faec12b9261fb363163bfa2f0ec770a2a6bce831fd2ac059c9 WatchSource:0}: Error finding container 0c5a6c41c3f495faec12b9261fb363163bfa2f0ec770a2a6bce831fd2ac059c9: Status 404 returned error can't find the container with id 0c5a6c41c3f495faec12b9261fb363163bfa2f0ec770a2a6bce831fd2ac059c9 Oct 31 13:19:06.555700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:06.555617 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" event={"ID":"ba157151-94be-49d2-a2ba-7a0e84091f3a","Type":"ContainerStarted","Data":"aefb70e0fe79bac6a84c6db1a3bbd9d15daa3992b60e79c5414eae73fd5fda25"} Oct 31 13:19:06.556295 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:06.556248 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" event={"ID":"ba157151-94be-49d2-a2ba-7a0e84091f3a","Type":"ContainerStarted","Data":"0c5a6c41c3f495faec12b9261fb363163bfa2f0ec770a2a6bce831fd2ac059c9"} Oct 31 13:19:06.579383 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:06.579334 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" podStartSLOduration=2.5793217029999997 podStartE2EDuration="2.579321703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:06.577671378 +0000 UTC m=+2840.755389362" watchObservedRunningTime="2026-10-31 13:19:06.579321703 +0000 UTC m=+2840.757039685" Oct 31 13:19:07.566387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:07.566354 5074 generic.go:356] "Generic (PLEG): container finished" podID="ba157151-94be-49d2-a2ba-7a0e84091f3a" containerID="aefb70e0fe79bac6a84c6db1a3bbd9d15daa3992b60e79c5414eae73fd5fda25" exitCode=0 Oct 31 13:19:07.566872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:07.566441 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" event={"ID":"ba157151-94be-49d2-a2ba-7a0e84091f3a","Type":"ContainerDied","Data":"aefb70e0fe79bac6a84c6db1a3bbd9d15daa3992b60e79c5414eae73fd5fda25"} Oct 31 13:19:09.117372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.117336 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72"] Oct 31 13:19:09.165942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.165914 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:09.201602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.201562 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba157151-94be-49d2-a2ba-7a0e84091f3a-kube-api-access-ttj5k\" (UniqueName: \"kubernetes.io/projected/ba157151-94be-49d2-a2ba-7a0e84091f3a-kube-api-access-ttj5k\") pod \"ba157151-94be-49d2-a2ba-7a0e84091f3a\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " Oct 31 13:19:09.201845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.201697 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-openshift-ca\") pod \"ba157151-94be-49d2-a2ba-7a0e84091f3a\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " Oct 31 13:19:09.201939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.201916 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-xks-ca\") pod \"ba157151-94be-49d2-a2ba-7a0e84091f3a\" (UID: \"ba157151-94be-49d2-a2ba-7a0e84091f3a\") " Oct 31 13:19:09.204096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.204065 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-openshift-ca" pod "ba157151-94be-49d2-a2ba-7a0e84091f3a" (UID: "ba157151-94be-49d2-a2ba-7a0e84091f3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:09.205567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.205543 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba157151-94be-49d2-a2ba-7a0e84091f3a-kube-api-access-ttj5k" pod "ba157151-94be-49d2-a2ba-7a0e84091f3a" (UID: "ba157151-94be-49d2-a2ba-7a0e84091f3a"). InnerVolumeSpecName "kube-api-access-ttj5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:09.212082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.212050 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-xks-ca" pod "ba157151-94be-49d2-a2ba-7a0e84091f3a" (UID: "ba157151-94be-49d2-a2ba-7a0e84091f3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:19:09.304463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.304419 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:09.304463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.304457 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttj5k\" (UniqueName: \"kubernetes.io/projected/ba157151-94be-49d2-a2ba-7a0e84091f3a-kube-api-access-ttj5k\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:09.304463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.304471 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba157151-94be-49d2-a2ba-7a0e84091f3a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:09.586008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.585964 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" event={"ID":"ba157151-94be-49d2-a2ba-7a0e84091f3a","Type":"ContainerDied","Data":"0c5a6c41c3f495faec12b9261fb363163bfa2f0ec770a2a6bce831fd2ac059c9"} Oct 31 13:19:09.586008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.586007 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g" Oct 31 13:19:09.586253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:09.586031 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5a6c41c3f495faec12b9261fb363163bfa2f0ec770a2a6bce831fd2ac059c9" Oct 31 13:19:12.410138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:12.410101 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx"] Oct 31 13:19:12.410620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:12.410309 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" containerID="cri-o://b4e65cfd2d3ed753425c46b86a735de2aea7c6844886ee1bffd5893e17b8eb05" gracePeriod=30 Oct 31 13:19:14.118679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.118576 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" podUID="eee85e49-07b8-4e0c-bb60-b9e7250b2214" containerName="payload-processing" containerID="cri-o://4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606" gracePeriod=25 Oct 31 13:19:14.425085 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.425058 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:19:14.509656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.506158 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-routing-overlay-volume\") pod \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " Oct 31 13:19:14.509656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.506287 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-kube-api-access-vd4xl\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-kube-api-access-vd4xl\") pod \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " Oct 31 13:19:14.509656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.506345 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-plugins-config-volume\") pod \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " Oct 31 13:19:14.509656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.506621 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-provider-credentials\") pod \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\" (UID: \"eee85e49-07b8-4e0c-bb60-b9e7250b2214\") " Oct 31 13:19:14.510069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.509888 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-routing-overlay-volume" pod "eee85e49-07b8-4e0c-bb60-b9e7250b2214" (UID: "eee85e49-07b8-4e0c-bb60-b9e7250b2214"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:14.510069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.510005 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-plugins-config-volume" pod "eee85e49-07b8-4e0c-bb60-b9e7250b2214" (UID: "eee85e49-07b8-4e0c-bb60-b9e7250b2214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:14.514884 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.514824 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-kube-api-access-vd4xl" pod "eee85e49-07b8-4e0c-bb60-b9e7250b2214" (UID: "eee85e49-07b8-4e0c-bb60-b9e7250b2214"). InnerVolumeSpecName "kube-api-access-vd4xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:14.515286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.515249 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-provider-credentials" pod "eee85e49-07b8-4e0c-bb60-b9e7250b2214" (UID: "eee85e49-07b8-4e0c-bb60-b9e7250b2214"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:14.611456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.611416 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:14.611456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.611456 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4xl\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-kube-api-access-vd4xl\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:14.611719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.611471 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee85e49-07b8-4e0c-bb60-b9e7250b2214-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:14.611719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.611487 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eee85e49-07b8-4e0c-bb60-b9e7250b2214-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:14.631531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.631442 5074 generic.go:356] "Generic (PLEG): container finished" podID="eee85e49-07b8-4e0c-bb60-b9e7250b2214" containerID="4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606" exitCode=0 Oct 31 13:19:14.631531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.631495 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" Oct 31 13:19:14.631811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.631531 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" event={"ID":"eee85e49-07b8-4e0c-bb60-b9e7250b2214","Type":"ContainerDied","Data":"4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606"} Oct 31 13:19:14.631811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.631591 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72" event={"ID":"eee85e49-07b8-4e0c-bb60-b9e7250b2214","Type":"ContainerDied","Data":"3a4ccf09f4e84048a6d27cd1bdc0c50f4c358ddf46a901fb52575394650b883f"} Oct 31 13:19:14.631811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.631616 5074 scope.go:122] "RemoveContainer" containerID="4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606" Oct 31 13:19:14.645496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.645475 5074 scope.go:122] "RemoveContainer" containerID="4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606" Oct 31 13:19:14.645889 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:19:14.645863 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606\": container with ID starting with 4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606 not found: ID does not exist" containerID="4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606" Oct 31 13:19:14.646030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.645894 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606"} err="failed to get container status \"4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606\": rpc error: code = NotFound desc = could not find container \"4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606\": container with ID starting with 4f9ceb63aabaaadb73e60f3c7ed41e4f9fc97df0676ef38b923a0be07fcbc606 not found: ID does not exist" Oct 31 13:19:14.664602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.664546 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72"] Oct 31 13:19:14.668909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:14.668886 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-29e3a0/payload-processing-external-model-e2e-extmod-b-29e3a0-6945dgn72"] Oct 31 13:19:15.941172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:15.941139 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee85e49-07b8-4e0c-bb60-b9e7250b2214" path="/var/lib/kubelet/pods/eee85e49-07b8-4e0c-bb60-b9e7250b2214/volumes" Oct 31 13:19:16.772345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.771960 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq"] Oct 31 13:19:16.774160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.773156 5074 cpu_manager.go:414] "containerMap: removing container" podUID="eee85e49-07b8-4e0c-bb60-b9e7250b2214" containerName="payload-processing" Oct 31 13:19:16.774160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.773183 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee85e49-07b8-4e0c-bb60-b9e7250b2214" containerName="payload-processing" Oct 31 13:19:16.774160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.773222 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ba157151-94be-49d2-a2ba-7a0e84091f3a" containerName="revoke-keys" Oct 31 13:19:16.774160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.773230 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba157151-94be-49d2-a2ba-7a0e84091f3a" containerName="revoke-keys" Oct 31 13:19:16.774160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.773397 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee85e49-07b8-4e0c-bb60-b9e7250b2214" containerName="payload-processing" Oct 31 13:19:16.774160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.773417 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba157151-94be-49d2-a2ba-7a0e84091f3a" containerName="revoke-keys" Oct 31 13:19:16.784235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.780398 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.803717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.803683 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:19:16.803895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.803875 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:19:16.803966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.803898 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:19:16.804036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.803988 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:19:16.806336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.806300 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq"] Oct 31 13:19:16.853009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.852972 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.853180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.853072 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7gzs\" (UniqueName: \"kubernetes.io/projected/63b61e10-257e-4d14-a1f8-52488c352b34-kube-api-access-h7gzs\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.853299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.853221 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.954473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.954438 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.954908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.954514 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7gzs\" (UniqueName: \"kubernetes.io/projected/63b61e10-257e-4d14-a1f8-52488c352b34-kube-api-access-h7gzs\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.954908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.954585 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.955215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.955194 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.956833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.956807 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:16.967642 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:16.967604 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7gzs\" (UniqueName: \"kubernetes.io/projected/63b61e10-257e-4d14-a1f8-52488c352b34-kube-api-access-h7gzs\") pod \"maas-api-revoke-sub-c0069429-bdnsq\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:17.119924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.119834 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:17.417449 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:19:17.417413 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b61e10_257e_4d14_a1f8_52488c352b34.slice/crio-44406e4ccb4bcf3b93a086efdb4f3d3f0da9c01b25cafa0a05cd36f516b25039 WatchSource:0}: Error finding container 44406e4ccb4bcf3b93a086efdb4f3d3f0da9c01b25cafa0a05cd36f516b25039: Status 404 returned error can't find the container with id 44406e4ccb4bcf3b93a086efdb4f3d3f0da9c01b25cafa0a05cd36f516b25039 Oct 31 13:19:17.417606 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.417572 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq"] Oct 31 13:19:17.705171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.702218 5074 generic.go:356] "Generic (PLEG): container finished" podID="f6546233-b00c-44c9-9d2a-e3078540b475" containerID="b4e65cfd2d3ed753425c46b86a735de2aea7c6844886ee1bffd5893e17b8eb05" exitCode=0 Oct 31 13:19:17.705171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.702368 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" event={"ID":"f6546233-b00c-44c9-9d2a-e3078540b475","Type":"ContainerDied","Data":"b4e65cfd2d3ed753425c46b86a735de2aea7c6844886ee1bffd5893e17b8eb05"} Oct 31 13:19:17.705171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.704188 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" event={"ID":"63b61e10-257e-4d14-a1f8-52488c352b34","Type":"ContainerStarted","Data":"44406e4ccb4bcf3b93a086efdb4f3d3f0da9c01b25cafa0a05cd36f516b25039"} Oct 31 13:19:17.756674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.756579 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:19:17.876005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.875971 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6546233-b00c-44c9-9d2a-e3078540b475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6546233-b00c-44c9-9d2a-e3078540b475-istiod-ca-cert\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876033 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-kube-api-access-b4rxs\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-kube-api-access-b4rxs\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876093 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-socket\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876135 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-credential-socket\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876358 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876236 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-certs\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876409 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6546233-b00c-44c9-9d2a-e3078540b475-istiod-ca-cert" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:17.876682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876507 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-credential-socket" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:17.876682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876597 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-certs" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:17.876682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876614 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6546233-b00c-44c9-9d2a-e3078540b475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6546233-b00c-44c9-9d2a-e3078540b475-istio-podinfo\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876662 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-istio-token\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-istio-token\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876795 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-envoy\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.876939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.876900 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-data\") pod \"f6546233-b00c-44c9-9d2a-e3078540b475\" (UID: \"f6546233-b00c-44c9-9d2a-e3078540b475\") " Oct 31 13:19:17.877215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.877158 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-socket" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:17.877305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.877279 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-data" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:17.877961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.877941 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.878154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.878136 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6546233-b00c-44c9-9d2a-e3078540b475-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.878260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.878246 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.878382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.878370 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.878468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.878456 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.879477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.879449 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-envoy" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:19:17.880003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.879979 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-istio-token" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:17.880126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.880008 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-kube-api-access-b4rxs" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "kube-api-access-b4rxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:17.880909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.880881 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6546233-b00c-44c9-9d2a-e3078540b475-istio-podinfo" pod "f6546233-b00c-44c9-9d2a-e3078540b475" (UID: "f6546233-b00c-44c9-9d2a-e3078540b475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:19:17.979796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.979767 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6546233-b00c-44c9-9d2a-e3078540b475-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.980152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.979798 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rxs\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-kube-api-access-b4rxs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.980152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.979814 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6546233-b00c-44c9-9d2a-e3078540b475-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:17.980152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:17.979826 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6546233-b00c-44c9-9d2a-e3078540b475-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:18.714333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:18.714290 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" Oct 31 13:19:18.714333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:18.714321 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx" event={"ID":"f6546233-b00c-44c9-9d2a-e3078540b475","Type":"ContainerDied","Data":"57cbe2c726191d820a53802739ed13d5e3d35ea00f299668b6daa437b364a7b5"} Oct 31 13:19:18.714607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:18.714383 5074 scope.go:122] "RemoveContainer" containerID="b4e65cfd2d3ed753425c46b86a735de2aea7c6844886ee1bffd5893e17b8eb05" Oct 31 13:19:18.717037 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:18.717009 5074 generic.go:356] "Generic (PLEG): container finished" podID="63b61e10-257e-4d14-a1f8-52488c352b34" containerID="0ca2a03399bbce046b5476b1343115c602aa474679f4b88ec92aafb54255a6a1" exitCode=0 Oct 31 13:19:18.717263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:18.717041 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" event={"ID":"63b61e10-257e-4d14-a1f8-52488c352b34","Type":"ContainerDied","Data":"0ca2a03399bbce046b5476b1343115c602aa474679f4b88ec92aafb54255a6a1"} Oct 31 13:19:18.751134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:18.751053 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx"] Oct 31 13:19:18.760512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:18.760470 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-6m2lx"] Oct 31 13:19:19.923966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:19.923938 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:19.940643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:19.940602 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" path="/var/lib/kubelet/pods/f6546233-b00c-44c9-9d2a-e3078540b475/volumes" Oct 31 13:19:20.025437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.025357 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-xks-ca\") pod \"63b61e10-257e-4d14-a1f8-52488c352b34\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " Oct 31 13:19:20.025595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.025534 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-openshift-ca\") pod \"63b61e10-257e-4d14-a1f8-52488c352b34\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " Oct 31 13:19:20.025678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.025596 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b61e10-257e-4d14-a1f8-52488c352b34-kube-api-access-h7gzs\" (UniqueName: \"kubernetes.io/projected/63b61e10-257e-4d14-a1f8-52488c352b34-kube-api-access-h7gzs\") pod \"63b61e10-257e-4d14-a1f8-52488c352b34\" (UID: \"63b61e10-257e-4d14-a1f8-52488c352b34\") " Oct 31 13:19:20.025936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.025911 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-openshift-ca" pod "63b61e10-257e-4d14-a1f8-52488c352b34" (UID: "63b61e10-257e-4d14-a1f8-52488c352b34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:20.026953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.026916 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:20.027918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.027888 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-xks-ca" pod "63b61e10-257e-4d14-a1f8-52488c352b34" (UID: "63b61e10-257e-4d14-a1f8-52488c352b34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:19:20.028111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.028093 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b61e10-257e-4d14-a1f8-52488c352b34-kube-api-access-h7gzs" pod "63b61e10-257e-4d14-a1f8-52488c352b34" (UID: "63b61e10-257e-4d14-a1f8-52488c352b34"). InnerVolumeSpecName "kube-api-access-h7gzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:20.128695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.128662 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7gzs\" (UniqueName: \"kubernetes.io/projected/63b61e10-257e-4d14-a1f8-52488c352b34-kube-api-access-h7gzs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:20.128695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.128697 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b61e10-257e-4d14-a1f8-52488c352b34-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:20.736497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.736461 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" Oct 31 13:19:20.736497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.736492 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq" event={"ID":"63b61e10-257e-4d14-a1f8-52488c352b34","Type":"ContainerDied","Data":"44406e4ccb4bcf3b93a086efdb4f3d3f0da9c01b25cafa0a05cd36f516b25039"} Oct 31 13:19:20.736750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:20.736542 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44406e4ccb4bcf3b93a086efdb4f3d3f0da9c01b25cafa0a05cd36f516b25039" Oct 31 13:19:21.409894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:21.409858 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt"] Oct 31 13:19:25.986007 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:19:25.985961 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:19:26.412744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.411739 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" podUID="7949dc6f-fd7b-445d-9177-0aa1635e8a00" containerName="payload-processing" containerID="cri-o://90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f" gracePeriod=25 Oct 31 13:19:26.721277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.721248 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:19:26.846692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.846608 5074 generic.go:356] "Generic (PLEG): container finished" podID="7949dc6f-fd7b-445d-9177-0aa1635e8a00" containerID="90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f" exitCode=0 Oct 31 13:19:26.846853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.846693 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" event={"ID":"7949dc6f-fd7b-445d-9177-0aa1635e8a00","Type":"ContainerDied","Data":"90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f"} Oct 31 13:19:26.846853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.846676 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" Oct 31 13:19:26.846853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.846789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt" event={"ID":"7949dc6f-fd7b-445d-9177-0aa1635e8a00","Type":"ContainerDied","Data":"b33882bbbb85778fc6e22dddf5965393af5a4d18a6bcdcfd36a3e6cfb1991f86"} Oct 31 13:19:26.846853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.846804 5074 scope.go:122] "RemoveContainer" containerID="90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f" Oct 31 13:19:26.860701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.852406 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-provider-credentials\") pod \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " Oct 31 13:19:26.860701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.852668 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-routing-overlay-volume\") pod \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " Oct 31 13:19:26.860701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.852805 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-plugins-config-volume\") pod \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " Oct 31 13:19:26.860701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.853011 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-kube-api-access-9rpm5\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-kube-api-access-9rpm5\") pod \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\" (UID: \"7949dc6f-fd7b-445d-9177-0aa1635e8a00\") " Oct 31 13:19:26.860701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.859181 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-routing-overlay-volume" pod "7949dc6f-fd7b-445d-9177-0aa1635e8a00" (UID: "7949dc6f-fd7b-445d-9177-0aa1635e8a00"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:26.860701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.859458 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-plugins-config-volume" pod "7949dc6f-fd7b-445d-9177-0aa1635e8a00" (UID: "7949dc6f-fd7b-445d-9177-0aa1635e8a00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:26.862175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.862142 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-provider-credentials" pod "7949dc6f-fd7b-445d-9177-0aa1635e8a00" (UID: "7949dc6f-fd7b-445d-9177-0aa1635e8a00"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:26.862833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.862795 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-kube-api-access-9rpm5" pod "7949dc6f-fd7b-445d-9177-0aa1635e8a00" (UID: "7949dc6f-fd7b-445d-9177-0aa1635e8a00"). InnerVolumeSpecName "kube-api-access-9rpm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:26.897790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.897766 5074 scope.go:122] "RemoveContainer" containerID="90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f" Oct 31 13:19:26.898174 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:19:26.898145 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f\": container with ID starting with 90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f not found: ID does not exist" containerID="90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f" Oct 31 13:19:26.898292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.898181 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f"} err="failed to get container status \"90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f\": rpc error: code = NotFound desc = could not find container \"90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f\": container with ID starting with 90d1fb48c4620a308774851d16cbad891f990aefe5ee90e746862865e001fa8f not found: ID does not exist" Oct 31 13:19:26.955159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.955096 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:26.955159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.955125 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7949dc6f-fd7b-445d-9177-0aa1635e8a00-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:26.955159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.955136 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rpm5\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-kube-api-access-9rpm5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:26.955159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:26.955145 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7949dc6f-fd7b-445d-9177-0aa1635e8a00-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:27.174142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.174109 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt"] Oct 31 13:19:27.180241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.180214 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-7887j74lt"] Oct 31 13:19:27.349625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.349588 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm"] Oct 31 13:19:27.350415 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350393 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" Oct 31 13:19:27.350508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350415 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" Oct 31 13:19:27.350508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350452 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7949dc6f-fd7b-445d-9177-0aa1635e8a00" containerName="payload-processing" Oct 31 13:19:27.350508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350458 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7949dc6f-fd7b-445d-9177-0aa1635e8a00" containerName="payload-processing" Oct 31 13:19:27.350508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350479 5074 cpu_manager.go:414] "containerMap: removing container" podUID="63b61e10-257e-4d14-a1f8-52488c352b34" containerName="revoke-keys" Oct 31 13:19:27.350508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350486 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b61e10-257e-4d14-a1f8-52488c352b34" containerName="revoke-keys" Oct 31 13:19:27.350758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350626 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6546233-b00c-44c9-9d2a-e3078540b475" containerName="istio-proxy" Oct 31 13:19:27.350758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350657 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7949dc6f-fd7b-445d-9177-0aa1635e8a00" containerName="payload-processing" Oct 31 13:19:27.350758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.350667 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b61e10-257e-4d14-a1f8-52488c352b34" containerName="revoke-keys" Oct 31 13:19:27.355799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.355780 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.360665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.359790 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:19:27.360665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.359969 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:19:27.360665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.360075 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:19:27.360665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.360255 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:19:27.363245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.363219 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm"] Oct 31 13:19:27.465894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.465860 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.466112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.465909 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.466112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.466072 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llnlk\" (UniqueName: \"kubernetes.io/projected/4b91eeea-72bc-475a-8671-a8da81a6a1e5-kube-api-access-llnlk\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.567358 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.567321 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.567549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.567372 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.567549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.567436 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llnlk\" (UniqueName: \"kubernetes.io/projected/4b91eeea-72bc-475a-8671-a8da81a6a1e5-kube-api-access-llnlk\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.568199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.568172 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.569766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.569741 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.580017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.579990 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llnlk\" (UniqueName: \"kubernetes.io/projected/4b91eeea-72bc-475a-8671-a8da81a6a1e5-kube-api-access-llnlk\") pod \"maas-api-revoke-sub-f1d5a0d2-ddrpm\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.677250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.677147 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:27.942485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.942407 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7949dc6f-fd7b-445d-9177-0aa1635e8a00" path="/var/lib/kubelet/pods/7949dc6f-fd7b-445d-9177-0aa1635e8a00/volumes" Oct 31 13:19:27.963443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:27.963414 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm"] Oct 31 13:19:27.963596 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:19:27.963539 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b91eeea_72bc_475a_8671_a8da81a6a1e5.slice/crio-4b5b9201a61e0064d9becc9cc1263ea58d2ce717f97744c67b6299a2cf5fef40 WatchSource:0}: Error finding container 4b5b9201a61e0064d9becc9cc1263ea58d2ce717f97744c67b6299a2cf5fef40: Status 404 returned error can't find the container with id 4b5b9201a61e0064d9becc9cc1263ea58d2ce717f97744c67b6299a2cf5fef40 Oct 31 13:19:28.867818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:28.867769 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" event={"ID":"4b91eeea-72bc-475a-8671-a8da81a6a1e5","Type":"ContainerStarted","Data":"da7c2c20958615fd0104f303ae9aa41cfda3e151e7047d77c65455d87c28243c"} Oct 31 13:19:28.868289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:28.867839 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" event={"ID":"4b91eeea-72bc-475a-8671-a8da81a6a1e5","Type":"ContainerStarted","Data":"4b5b9201a61e0064d9becc9cc1263ea58d2ce717f97744c67b6299a2cf5fef40"} Oct 31 13:19:28.895729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:28.895659 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" podStartSLOduration=1.895626727 podStartE2EDuration="1.895626727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:28.892519377 +0000 UTC m=+2863.070237362" watchObservedRunningTime="2026-10-31 13:19:28.895626727 +0000 UTC m=+2863.073344711" Oct 31 13:19:29.880125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:29.880090 5074 generic.go:356] "Generic (PLEG): container finished" podID="4b91eeea-72bc-475a-8671-a8da81a6a1e5" containerID="da7c2c20958615fd0104f303ae9aa41cfda3e151e7047d77c65455d87c28243c" exitCode=0 Oct 31 13:19:29.880538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:29.880174 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" event={"ID":"4b91eeea-72bc-475a-8671-a8da81a6a1e5","Type":"ContainerDied","Data":"da7c2c20958615fd0104f303ae9aa41cfda3e151e7047d77c65455d87c28243c"} Oct 31 13:19:31.081096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.081064 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:31.234513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.234476 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-xks-ca\") pod \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " Oct 31 13:19:31.234716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.234520 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b91eeea-72bc-475a-8671-a8da81a6a1e5-kube-api-access-llnlk\" (UniqueName: \"kubernetes.io/projected/4b91eeea-72bc-475a-8671-a8da81a6a1e5-kube-api-access-llnlk\") pod \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " Oct 31 13:19:31.234774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.234750 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-openshift-ca\") pod \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\" (UID: \"4b91eeea-72bc-475a-8671-a8da81a6a1e5\") " Oct 31 13:19:31.235111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.235085 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-openshift-ca" pod "4b91eeea-72bc-475a-8671-a8da81a6a1e5" (UID: "4b91eeea-72bc-475a-8671-a8da81a6a1e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:31.235367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.235349 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:31.236734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.236699 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-xks-ca" pod "4b91eeea-72bc-475a-8671-a8da81a6a1e5" (UID: "4b91eeea-72bc-475a-8671-a8da81a6a1e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:19:31.236845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.236823 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b91eeea-72bc-475a-8671-a8da81a6a1e5-kube-api-access-llnlk" pod "4b91eeea-72bc-475a-8671-a8da81a6a1e5" (UID: "4b91eeea-72bc-475a-8671-a8da81a6a1e5"). InnerVolumeSpecName "kube-api-access-llnlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:31.336467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.336419 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b91eeea-72bc-475a-8671-a8da81a6a1e5-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:31.336467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.336461 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llnlk\" (UniqueName: \"kubernetes.io/projected/4b91eeea-72bc-475a-8671-a8da81a6a1e5-kube-api-access-llnlk\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:31.899291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.899250 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" Oct 31 13:19:31.899506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.899282 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm" event={"ID":"4b91eeea-72bc-475a-8671-a8da81a6a1e5","Type":"ContainerDied","Data":"4b5b9201a61e0064d9becc9cc1263ea58d2ce717f97744c67b6299a2cf5fef40"} Oct 31 13:19:31.899506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:31.899339 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5b9201a61e0064d9becc9cc1263ea58d2ce717f97744c67b6299a2cf5fef40" Oct 31 13:19:39.730035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.729984 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb"] Oct 31 13:19:39.730774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.730757 5074 cpu_manager.go:414] "containerMap: removing container" podUID="4b91eeea-72bc-475a-8671-a8da81a6a1e5" containerName="revoke-keys" Oct 31 13:19:39.730774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.730774 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b91eeea-72bc-475a-8671-a8da81a6a1e5" containerName="revoke-keys" Oct 31 13:19:39.730908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.730899 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b91eeea-72bc-475a-8671-a8da81a6a1e5" containerName="revoke-keys" Oct 31 13:19:39.734890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.734869 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.739920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.739891 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:19:39.740102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.740035 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:19:39.740196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.740098 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:19:39.740196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.740170 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:19:39.744142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.744110 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb"] Oct 31 13:19:39.823748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.823700 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.823937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.823913 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqf2\" (UniqueName: \"kubernetes.io/projected/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-kube-api-access-ckqf2\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.824150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.824130 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.926654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.925961 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.926654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.926114 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.926654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.926239 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqf2\" (UniqueName: \"kubernetes.io/projected/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-kube-api-access-ckqf2\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.940652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.927849 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.960906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.960867 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx"] Oct 31 13:19:39.969455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.969422 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqf2\" (UniqueName: \"kubernetes.io/projected/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-kube-api-access-ckqf2\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:39.987447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:39.987345 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c557a6-frbnb\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:40.012195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.012157 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx"] Oct 31 13:19:40.012443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.012274 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.055784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.055750 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:40.133495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.128884 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4wx\" (UniqueName: \"kubernetes.io/projected/53669b16-6aa1-4440-a00b-79156a01fcb9-kube-api-access-lg4wx\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.133495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.129179 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.133495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.129372 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.232336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.232288 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.232698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.232510 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.232698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.232558 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4wx\" (UniqueName: \"kubernetes.io/projected/53669b16-6aa1-4440-a00b-79156a01fcb9-kube-api-access-lg4wx\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.233720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.233690 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.236727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.236702 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.246682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.246605 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4wx\" (UniqueName: \"kubernetes.io/projected/53669b16-6aa1-4440-a00b-79156a01fcb9-kube-api-access-lg4wx\") pod \"maas-api-revoke-sub-4d652d47-dqqfx\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.329672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.329613 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:40.345086 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:19:40.345049 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca25edb_a97b_400c_b8fa_6cc2efdc6575.slice/crio-e5635c1d634e2772fad779b7baa2b229e759c88badf13c30f8b7261bbcae56d7 WatchSource:0}: Error finding container e5635c1d634e2772fad779b7baa2b229e759c88badf13c30f8b7261bbcae56d7: Status 404 returned error can't find the container with id e5635c1d634e2772fad779b7baa2b229e759c88badf13c30f8b7261bbcae56d7 Oct 31 13:19:40.345211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.345132 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb"] Oct 31 13:19:40.619023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:40.618993 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx"] Oct 31 13:19:40.619478 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:19:40.619453 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53669b16_6aa1_4440_a00b_79156a01fcb9.slice/crio-d4266ea3dbf424b1085a323c37868904f1272b585f74c3a6ab0b1f861a417189 WatchSource:0}: Error finding container d4266ea3dbf424b1085a323c37868904f1272b585f74c3a6ab0b1f861a417189: Status 404 returned error can't find the container with id d4266ea3dbf424b1085a323c37868904f1272b585f74c3a6ab0b1f861a417189 Oct 31 13:19:41.124815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:41.124770 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" event={"ID":"1ca25edb-a97b-400c-b8fa-6cc2efdc6575","Type":"ContainerStarted","Data":"b89471695e5b593e997f5367ed1558c07c29d0e4cb6b9beccae450ba37fc2430"} Oct 31 13:19:41.125333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:41.124873 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" event={"ID":"1ca25edb-a97b-400c-b8fa-6cc2efdc6575","Type":"ContainerStarted","Data":"e5635c1d634e2772fad779b7baa2b229e759c88badf13c30f8b7261bbcae56d7"} Oct 31 13:19:41.126500 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:41.126465 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" event={"ID":"53669b16-6aa1-4440-a00b-79156a01fcb9","Type":"ContainerStarted","Data":"a07012d460debad3f1951cee9611fb7166e8fa42cc88d76e556aba068033c31c"} Oct 31 13:19:41.126662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:41.126519 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" event={"ID":"53669b16-6aa1-4440-a00b-79156a01fcb9","Type":"ContainerStarted","Data":"d4266ea3dbf424b1085a323c37868904f1272b585f74c3a6ab0b1f861a417189"} Oct 31 13:19:41.188077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:41.187972 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" podStartSLOduration=2.187957283 podStartE2EDuration="2.187957283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:41.183144367 +0000 UTC m=+2875.360862352" watchObservedRunningTime="2026-10-31 13:19:41.187957283 +0000 UTC m=+2875.365675266" Oct 31 13:19:41.226481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:41.226424 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" podStartSLOduration=2.226410052 podStartE2EDuration="2.226410052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:41.221752474 +0000 UTC m=+2875.399470462" watchObservedRunningTime="2026-10-31 13:19:41.226410052 +0000 UTC m=+2875.404128063" Oct 31 13:19:42.137557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:42.137516 5074 generic.go:356] "Generic (PLEG): container finished" podID="1ca25edb-a97b-400c-b8fa-6cc2efdc6575" containerID="b89471695e5b593e997f5367ed1558c07c29d0e4cb6b9beccae450ba37fc2430" exitCode=0 Oct 31 13:19:42.138111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:42.137596 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" event={"ID":"1ca25edb-a97b-400c-b8fa-6cc2efdc6575","Type":"ContainerDied","Data":"b89471695e5b593e997f5367ed1558c07c29d0e4cb6b9beccae450ba37fc2430"} Oct 31 13:19:42.139694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:42.139616 5074 generic.go:356] "Generic (PLEG): container finished" podID="53669b16-6aa1-4440-a00b-79156a01fcb9" containerID="a07012d460debad3f1951cee9611fb7166e8fa42cc88d76e556aba068033c31c" exitCode=0 Oct 31 13:19:42.139821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:42.139667 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" event={"ID":"53669b16-6aa1-4440-a00b-79156a01fcb9","Type":"ContainerDied","Data":"a07012d460debad3f1951cee9611fb7166e8fa42cc88d76e556aba068033c31c"} Oct 31 13:19:43.351586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.351557 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:43.384192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.384157 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-kube-api-access-ckqf2\" (UniqueName: \"kubernetes.io/projected/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-kube-api-access-ckqf2\") pod \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " Oct 31 13:19:43.384385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.384350 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-xks-ca\") pod \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " Oct 31 13:19:43.384463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.384421 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-openshift-ca\") pod \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\" (UID: \"1ca25edb-a97b-400c-b8fa-6cc2efdc6575\") " Oct 31 13:19:43.385569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.385536 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-openshift-ca" pod "1ca25edb-a97b-400c-b8fa-6cc2efdc6575" (UID: "1ca25edb-a97b-400c-b8fa-6cc2efdc6575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:43.385982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.385959 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:43.390908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.390877 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-xks-ca" pod "1ca25edb-a97b-400c-b8fa-6cc2efdc6575" (UID: "1ca25edb-a97b-400c-b8fa-6cc2efdc6575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:19:43.392549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.392209 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-kube-api-access-ckqf2" pod "1ca25edb-a97b-400c-b8fa-6cc2efdc6575" (UID: "1ca25edb-a97b-400c-b8fa-6cc2efdc6575"). InnerVolumeSpecName "kube-api-access-ckqf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:43.392549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.392225 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:43.495525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.495023 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckqf2\" (UniqueName: \"kubernetes.io/projected/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-kube-api-access-ckqf2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:43.495525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.495067 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca25edb-a97b-400c-b8fa-6cc2efdc6575-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:43.596156 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.596110 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53669b16-6aa1-4440-a00b-79156a01fcb9-kube-api-access-lg4wx\" (UniqueName: \"kubernetes.io/projected/53669b16-6aa1-4440-a00b-79156a01fcb9-kube-api-access-lg4wx\") pod \"53669b16-6aa1-4440-a00b-79156a01fcb9\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " Oct 31 13:19:43.596370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.596186 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-openshift-ca\") pod \"53669b16-6aa1-4440-a00b-79156a01fcb9\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " Oct 31 13:19:43.596370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.596321 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-xks-ca\") pod \"53669b16-6aa1-4440-a00b-79156a01fcb9\" (UID: \"53669b16-6aa1-4440-a00b-79156a01fcb9\") " Oct 31 13:19:43.596692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.596661 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-openshift-ca" pod "53669b16-6aa1-4440-a00b-79156a01fcb9" (UID: "53669b16-6aa1-4440-a00b-79156a01fcb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:43.597274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.597246 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:43.598938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.598903 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53669b16-6aa1-4440-a00b-79156a01fcb9-kube-api-access-lg4wx" pod "53669b16-6aa1-4440-a00b-79156a01fcb9" (UID: "53669b16-6aa1-4440-a00b-79156a01fcb9"). InnerVolumeSpecName "kube-api-access-lg4wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:43.599027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.598946 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-xks-ca" pod "53669b16-6aa1-4440-a00b-79156a01fcb9" (UID: "53669b16-6aa1-4440-a00b-79156a01fcb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:19:43.699207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.699114 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4wx\" (UniqueName: \"kubernetes.io/projected/53669b16-6aa1-4440-a00b-79156a01fcb9-kube-api-access-lg4wx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:43.699207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:43.699151 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53669b16-6aa1-4440-a00b-79156a01fcb9-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:44.160958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:44.160924 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" Oct 31 13:19:44.160958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:44.160950 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb" event={"ID":"1ca25edb-a97b-400c-b8fa-6cc2efdc6575","Type":"ContainerDied","Data":"e5635c1d634e2772fad779b7baa2b229e759c88badf13c30f8b7261bbcae56d7"} Oct 31 13:19:44.161191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:44.161001 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5635c1d634e2772fad779b7baa2b229e759c88badf13c30f8b7261bbcae56d7" Oct 31 13:19:44.162591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:44.162556 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" event={"ID":"53669b16-6aa1-4440-a00b-79156a01fcb9","Type":"ContainerDied","Data":"d4266ea3dbf424b1085a323c37868904f1272b585f74c3a6ab0b1f861a417189"} Oct 31 13:19:44.162774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:44.162605 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4266ea3dbf424b1085a323c37868904f1272b585f74c3a6ab0b1f861a417189" Oct 31 13:19:44.162774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:44.162619 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx" Oct 31 13:19:51.283661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.283336 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh"] Oct 31 13:19:51.302658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.284489 5074 cpu_manager.go:414] "containerMap: removing container" podUID="53669b16-6aa1-4440-a00b-79156a01fcb9" containerName="revoke-keys" Oct 31 13:19:51.302658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.284516 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53669b16-6aa1-4440-a00b-79156a01fcb9" containerName="revoke-keys" Oct 31 13:19:51.302658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.284536 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1ca25edb-a97b-400c-b8fa-6cc2efdc6575" containerName="revoke-keys" Oct 31 13:19:51.302658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.284544 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca25edb-a97b-400c-b8fa-6cc2efdc6575" containerName="revoke-keys" Oct 31 13:19:51.302658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.284769 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="53669b16-6aa1-4440-a00b-79156a01fcb9" containerName="revoke-keys" Oct 31 13:19:51.302658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.284789 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca25edb-a97b-400c-b8fa-6cc2efdc6575" containerName="revoke-keys" Oct 31 13:19:51.312656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.310008 5074 scope.go:122] "RemoveContainer" containerID="e3db449cac2291f70952d965b9ebf3149cbfc33bc56fec70089b5f9007532efc" Oct 31 13:19:51.541731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.541465 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh"] Oct 31 13:19:51.541731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.541604 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.553326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.551081 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2ef686\"" Oct 31 13:19:51.553326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.551247 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"routing-overlay\"" Oct 31 13:19:51.553326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.551772 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"openshift-service-ca.crt\"" Oct 31 13:19:51.553326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.552000 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"payload-processing-external-model-e2e--362fde4e-dockercfg-l7p2q\"" Oct 31 13:19:51.553326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.552156 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"kube-root-ca.crt\"" Oct 31 13:19:51.553326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.552261 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2ef686\"/\"agc-cred-811c6a0d5a4c\"" Oct 31 13:19:51.749659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.724657 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.750033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.749982 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.750240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.750222 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722zr\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-kube-api-access-722zr\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.750647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.750610 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.864342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.822999 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc"] Oct 31 13:19:51.864342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.855722 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.864342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.855804 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722zr\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-kube-api-access-722zr\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.864342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.856010 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.864342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.856162 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.864342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.857247 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.864342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.857833 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.920449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.913835 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:51.929482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.922509 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:51.952943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.952912 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:19:51.953904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.953815 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:19:51.954198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.954181 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:19:51.954445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.954428 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:19:51.992664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:51.978546 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722zr\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-kube-api-access-722zr\") pod \"payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:52.054415 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.054384 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc"] Oct 31 13:19:52.063552 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.063521 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.063767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.063741 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.064260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.064224 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9c29\" (UniqueName: \"kubernetes.io/projected/dcb8e7cb-2a7e-4506-858e-1c4689817e03-kube-api-access-b9c29\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.165508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.165416 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9c29\" (UniqueName: \"kubernetes.io/projected/dcb8e7cb-2a7e-4506-858e-1c4689817e03-kube-api-access-b9c29\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.165715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.165541 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.165715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.165659 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.166234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.166207 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.168003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.167975 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.177694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.177647 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:52.178670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.178615 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9c29\" (UniqueName: \"kubernetes.io/projected/dcb8e7cb-2a7e-4506-858e-1c4689817e03-kube-api-access-b9c29\") pod \"maas-api-revoke-sub-22c0f652-26ljc\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.352871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.352836 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:52.482488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.481978 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh"] Oct 31 13:19:52.675188 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:19:52.675150 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb8e7cb_2a7e_4506_858e_1c4689817e03.slice/crio-e755b987b5121463909de6348b7916fc6188919bf19089ad986b9ab8a3a85709 WatchSource:0}: Error finding container e755b987b5121463909de6348b7916fc6188919bf19089ad986b9ab8a3a85709: Status 404 returned error can't find the container with id e755b987b5121463909de6348b7916fc6188919bf19089ad986b9ab8a3a85709 Oct 31 13:19:52.675368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:52.675339 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc"] Oct 31 13:19:53.345344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:53.345255 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" event={"ID":"0f191422-4eb5-4b03-9163-3492c06004a4","Type":"ContainerStarted","Data":"4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14"} Oct 31 13:19:53.345344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:53.345316 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" event={"ID":"0f191422-4eb5-4b03-9163-3492c06004a4","Type":"ContainerStarted","Data":"fe51dc23ad2d68a037d84a71becb1f98e6f773cf4dd40c0360d65405b9ef5210"} Oct 31 13:19:53.345344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:53.345342 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:19:53.347252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:53.347220 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" event={"ID":"dcb8e7cb-2a7e-4506-858e-1c4689817e03","Type":"ContainerStarted","Data":"b6e4850ba1f5b3cd9b76d88f3f3f2dc21e154ca093e89e9ca1ff9b92842aa7ee"} Oct 31 13:19:53.347381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:53.347273 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" event={"ID":"dcb8e7cb-2a7e-4506-858e-1c4689817e03","Type":"ContainerStarted","Data":"e755b987b5121463909de6348b7916fc6188919bf19089ad986b9ab8a3a85709"} Oct 31 13:19:53.371700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:53.371644 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" podStartSLOduration=2.371617479 podStartE2EDuration="2.371617479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:53.366373328 +0000 UTC m=+2887.544091307" watchObservedRunningTime="2026-10-31 13:19:53.371617479 +0000 UTC m=+2887.549335462" Oct 31 13:19:53.392731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:53.392626 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" podStartSLOduration=2.392612162 podStartE2EDuration="2.392612162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:53.388961312 +0000 UTC m=+2887.566679297" watchObservedRunningTime="2026-10-31 13:19:53.392612162 +0000 UTC m=+2887.570330146" Oct 31 13:19:54.356130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:54.356088 5074 generic.go:356] "Generic (PLEG): container finished" podID="dcb8e7cb-2a7e-4506-858e-1c4689817e03" containerID="b6e4850ba1f5b3cd9b76d88f3f3f2dc21e154ca093e89e9ca1ff9b92842aa7ee" exitCode=0 Oct 31 13:19:54.356332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:54.356172 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" event={"ID":"dcb8e7cb-2a7e-4506-858e-1c4689817e03","Type":"ContainerDied","Data":"b6e4850ba1f5b3cd9b76d88f3f3f2dc21e154ca093e89e9ca1ff9b92842aa7ee"} Oct 31 13:19:55.635248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.635182 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:55.735723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.735688 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-xks-ca\") pod \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " Oct 31 13:19:55.735934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.735831 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb8e7cb-2a7e-4506-858e-1c4689817e03-kube-api-access-b9c29\" (UniqueName: \"kubernetes.io/projected/dcb8e7cb-2a7e-4506-858e-1c4689817e03-kube-api-access-b9c29\") pod \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " Oct 31 13:19:55.736041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.736026 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-openshift-ca\") pod \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\" (UID: \"dcb8e7cb-2a7e-4506-858e-1c4689817e03\") " Oct 31 13:19:55.736337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.736315 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-openshift-ca" pod "dcb8e7cb-2a7e-4506-858e-1c4689817e03" (UID: "dcb8e7cb-2a7e-4506-858e-1c4689817e03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:19:55.736622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.736606 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:55.737837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.737812 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-xks-ca" pod "dcb8e7cb-2a7e-4506-858e-1c4689817e03" (UID: "dcb8e7cb-2a7e-4506-858e-1c4689817e03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:19:55.737984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.737964 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb8e7cb-2a7e-4506-858e-1c4689817e03-kube-api-access-b9c29" pod "dcb8e7cb-2a7e-4506-858e-1c4689817e03" (UID: "dcb8e7cb-2a7e-4506-858e-1c4689817e03"). InnerVolumeSpecName "kube-api-access-b9c29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:19:55.837691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.837621 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9c29\" (UniqueName: \"kubernetes.io/projected/dcb8e7cb-2a7e-4506-858e-1c4689817e03-kube-api-access-b9c29\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:55.837691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:55.837689 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb8e7cb-2a7e-4506-858e-1c4689817e03-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:19:55.984400 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:19:55.984358 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:19:56.392944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:56.392914 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" Oct 31 13:19:56.392944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:56.392930 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc" event={"ID":"dcb8e7cb-2a7e-4506-858e-1c4689817e03","Type":"ContainerDied","Data":"e755b987b5121463909de6348b7916fc6188919bf19089ad986b9ab8a3a85709"} Oct 31 13:19:56.393236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:56.392988 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e755b987b5121463909de6348b7916fc6188919bf19089ad986b9ab8a3a85709" Oct 31 13:19:56.620559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:19:56.620521 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:20:00.873736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.873694 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4"] Oct 31 13:20:00.875027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.874768 5074 cpu_manager.go:414] "containerMap: removing container" podUID="dcb8e7cb-2a7e-4506-858e-1c4689817e03" containerName="revoke-keys" Oct 31 13:20:00.875027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.874791 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb8e7cb-2a7e-4506-858e-1c4689817e03" containerName="revoke-keys" Oct 31 13:20:00.875027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.875007 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb8e7cb-2a7e-4506-858e-1c4689817e03" containerName="revoke-keys" Oct 31 13:20:00.879286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.879242 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:00.883835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.883809 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:20:00.883956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.883885 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:20:00.884584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.884565 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:20:00.884939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.884915 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:20:00.885939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:00.885916 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4"] Oct 31 13:20:01.002338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.002288 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.002583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.002558 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75x9r\" (UniqueName: \"kubernetes.io/projected/4a2716d1-f401-47c2-a7e8-cfe56628a063-kube-api-access-75x9r\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.002873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.002750 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.104810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.104760 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75x9r\" (UniqueName: \"kubernetes.io/projected/4a2716d1-f401-47c2-a7e8-cfe56628a063-kube-api-access-75x9r\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.104998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.104870 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.105060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.105044 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.105734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.105705 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.107527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.107499 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.117843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.117819 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75x9r\" (UniqueName: \"kubernetes.io/projected/4a2716d1-f401-47c2-a7e8-cfe56628a063-kube-api-access-75x9r\") pod \"maas-api-revoke-sub-e5b4f77c-zwgw4\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.199538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.199452 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:01.494391 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:20:01.494345 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2716d1_f401_47c2_a7e8_cfe56628a063.slice/crio-ca8c504bfc728baf9e05e192d28418b834ae0fa8dcf69efa74e38e36182e7859 WatchSource:0}: Error finding container ca8c504bfc728baf9e05e192d28418b834ae0fa8dcf69efa74e38e36182e7859: Status 404 returned error can't find the container with id ca8c504bfc728baf9e05e192d28418b834ae0fa8dcf69efa74e38e36182e7859 Oct 31 13:20:01.494599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:01.494575 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4"] Oct 31 13:20:02.489366 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:02.489329 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" event={"ID":"4a2716d1-f401-47c2-a7e8-cfe56628a063","Type":"ContainerStarted","Data":"a4b0d72db7788417563fd72d43685370f0db06483dd5bd1d0547e6797f37a965"} Oct 31 13:20:02.489911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:02.489400 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" event={"ID":"4a2716d1-f401-47c2-a7e8-cfe56628a063","Type":"ContainerStarted","Data":"ca8c504bfc728baf9e05e192d28418b834ae0fa8dcf69efa74e38e36182e7859"} Oct 31 13:20:02.513902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:02.513827 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" podStartSLOduration=2.513813492 podStartE2EDuration="2.513813492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:02.508530265 +0000 UTC m=+2896.686248249" watchObservedRunningTime="2026-10-31 13:20:02.513813492 +0000 UTC m=+2896.691531475" Oct 31 13:20:03.505704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:03.504752 5074 generic.go:356] "Generic (PLEG): container finished" podID="4a2716d1-f401-47c2-a7e8-cfe56628a063" containerID="a4b0d72db7788417563fd72d43685370f0db06483dd5bd1d0547e6797f37a965" exitCode=0 Oct 31 13:20:03.505704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:03.504855 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" event={"ID":"4a2716d1-f401-47c2-a7e8-cfe56628a063","Type":"ContainerDied","Data":"a4b0d72db7788417563fd72d43685370f0db06483dd5bd1d0547e6797f37a965"} Oct 31 13:20:03.790826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:03.790739 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh"] Oct 31 13:20:03.793922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:03.793896 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:20:04.702899 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.702864 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:04.842098 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.842009 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-openshift-ca\") pod \"4a2716d1-f401-47c2-a7e8-cfe56628a063\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " Oct 31 13:20:04.842280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.842174 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-xks-ca\") pod \"4a2716d1-f401-47c2-a7e8-cfe56628a063\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " Oct 31 13:20:04.842280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.842261 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2716d1-f401-47c2-a7e8-cfe56628a063-kube-api-access-75x9r\" (UniqueName: \"kubernetes.io/projected/4a2716d1-f401-47c2-a7e8-cfe56628a063-kube-api-access-75x9r\") pod \"4a2716d1-f401-47c2-a7e8-cfe56628a063\" (UID: \"4a2716d1-f401-47c2-a7e8-cfe56628a063\") " Oct 31 13:20:04.842472 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.842445 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-openshift-ca" pod "4a2716d1-f401-47c2-a7e8-cfe56628a063" (UID: "4a2716d1-f401-47c2-a7e8-cfe56628a063"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:04.843042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.843017 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:04.844389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.844366 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2716d1-f401-47c2-a7e8-cfe56628a063-kube-api-access-75x9r" pod "4a2716d1-f401-47c2-a7e8-cfe56628a063" (UID: "4a2716d1-f401-47c2-a7e8-cfe56628a063"). InnerVolumeSpecName "kube-api-access-75x9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:04.844494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.844368 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-xks-ca" pod "4a2716d1-f401-47c2-a7e8-cfe56628a063" (UID: "4a2716d1-f401-47c2-a7e8-cfe56628a063"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:20:04.945253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.945209 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2716d1-f401-47c2-a7e8-cfe56628a063-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:04.945253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:04.945255 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75x9r\" (UniqueName: \"kubernetes.io/projected/4a2716d1-f401-47c2-a7e8-cfe56628a063-kube-api-access-75x9r\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:05.528960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:05.528927 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" Oct 31 13:20:05.528960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:05.528942 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4" event={"ID":"4a2716d1-f401-47c2-a7e8-cfe56628a063","Type":"ContainerDied","Data":"ca8c504bfc728baf9e05e192d28418b834ae0fa8dcf69efa74e38e36182e7859"} Oct 31 13:20:05.531232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:05.529000 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca8c504bfc728baf9e05e192d28418b834ae0fa8dcf69efa74e38e36182e7859" Oct 31 13:20:08.112668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.109582 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8"] Oct 31 13:20:08.112668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.110751 5074 cpu_manager.go:414] "containerMap: removing container" podUID="4a2716d1-f401-47c2-a7e8-cfe56628a063" containerName="revoke-keys" Oct 31 13:20:08.112668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.110776 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2716d1-f401-47c2-a7e8-cfe56628a063" containerName="revoke-keys" Oct 31 13:20:08.112668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.111012 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2716d1-f401-47c2-a7e8-cfe56628a063" containerName="revoke-keys" Oct 31 13:20:08.125206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.116706 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.141611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.141227 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"agc-cred-225017692890\"" Oct 31 13:20:08.141611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.141352 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"routing-overlay\"" Oct 31 13:20:08.141611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.141447 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"openshift-service-ca.crt\"" Oct 31 13:20:08.142202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.142114 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"kube-root-ca.crt\"" Oct 31 13:20:08.142202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.142169 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ef744\"" Oct 31 13:20:08.145958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.145924 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8"] Oct 31 13:20:08.308689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.260691 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.308689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.260780 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.308689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.260841 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.308689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.260920 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gx6\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-kube-api-access-s2gx6\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.362579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.362532 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.362811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.362624 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.362811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.362717 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.362811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.362794 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gx6\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-kube-api-access-s2gx6\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.363522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.363492 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.363786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.363669 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.365468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.365434 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.386652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.376036 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gx6\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-kube-api-access-s2gx6\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.510125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.499062 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:08.536672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.533746 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt"] Oct 31 13:20:08.600976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.600933 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt"] Oct 31 13:20:08.610812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.610770 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.633722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.632148 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"agc-cred-1a57b50b6129\"" Oct 31 13:20:08.633722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.632279 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"payload-processing-external-model-e2e--4724aaf0-dockercfg-fj5fx\"" Oct 31 13:20:08.778970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.774675 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.778970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.774915 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.778970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.775008 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.778970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.775107 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68pm\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-kube-api-access-q68pm\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.803067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.803007 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" podUID="0f191422-4eb5-4b03-9163-3492c06004a4" containerName="payload-processing" containerID="cri-o://4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14" gracePeriod=25 Oct 31 13:20:08.882287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.880470 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.882287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.880591 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.882287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.880697 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68pm\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-kube-api-access-q68pm\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.882287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.880801 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.882287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.881847 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.886680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.886612 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.894023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.893819 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:08.935446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:08.935377 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68pm\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-kube-api-access-q68pm\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:09.139769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.139729 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8"] Oct 31 13:20:09.142247 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:20:09.142218 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c1e4d_8aa2_4be3_b773_b12b547aa0a0.slice/crio-be0442fcc3ed93197f51bbc002c7402cfa6919caae54f5fe613f1cfd1169ab8f WatchSource:0}: Error finding container be0442fcc3ed93197f51bbc002c7402cfa6919caae54f5fe613f1cfd1169ab8f: Status 404 returned error can't find the container with id be0442fcc3ed93197f51bbc002c7402cfa6919caae54f5fe613f1cfd1169ab8f Oct 31 13:20:09.143325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.143249 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:09.179417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.179007 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:20:09.293277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.293226 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-kube-api-access-722zr\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-kube-api-access-722zr\") pod \"0f191422-4eb5-4b03-9163-3492c06004a4\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " Oct 31 13:20:09.293479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.293428 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-provider-credentials\") pod \"0f191422-4eb5-4b03-9163-3492c06004a4\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " Oct 31 13:20:09.293559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.293537 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-plugins-config-volume\") pod \"0f191422-4eb5-4b03-9163-3492c06004a4\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " Oct 31 13:20:09.293660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.293624 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-routing-overlay-volume\") pod \"0f191422-4eb5-4b03-9163-3492c06004a4\" (UID: \"0f191422-4eb5-4b03-9163-3492c06004a4\") " Oct 31 13:20:09.295778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.295745 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-plugins-config-volume" pod "0f191422-4eb5-4b03-9163-3492c06004a4" (UID: "0f191422-4eb5-4b03-9163-3492c06004a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:09.296207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.296180 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-routing-overlay-volume" pod "0f191422-4eb5-4b03-9163-3492c06004a4" (UID: "0f191422-4eb5-4b03-9163-3492c06004a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:09.299916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.299877 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-kube-api-access-722zr" pod "0f191422-4eb5-4b03-9163-3492c06004a4" (UID: "0f191422-4eb5-4b03-9163-3492c06004a4"). InnerVolumeSpecName "kube-api-access-722zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:09.301557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.301534 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-provider-credentials" pod "0f191422-4eb5-4b03-9163-3492c06004a4" (UID: "0f191422-4eb5-4b03-9163-3492c06004a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:09.395907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.395803 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722zr\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-kube-api-access-722zr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:09.395907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.395838 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f191422-4eb5-4b03-9163-3492c06004a4-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:09.395907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.395854 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:09.395907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.395866 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f191422-4eb5-4b03-9163-3492c06004a4-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:09.476660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.471475 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt"] Oct 31 13:20:09.519159 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:20:09.519127 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57421ac_c611_4bee_a0fe_a7e4b143b00a.slice/crio-61df2449b4b9fb89648cbd8adcf8906580cf55402a9409674437d3d59a4a8b91 WatchSource:0}: Error finding container 61df2449b4b9fb89648cbd8adcf8906580cf55402a9409674437d3d59a4a8b91: Status 404 returned error can't find the container with id 61df2449b4b9fb89648cbd8adcf8906580cf55402a9409674437d3d59a4a8b91 Oct 31 13:20:09.829838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.829801 5074 generic.go:356] "Generic (PLEG): container finished" podID="0f191422-4eb5-4b03-9163-3492c06004a4" containerID="4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14" exitCode=0 Oct 31 13:20:09.830045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.829877 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" Oct 31 13:20:09.830045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.829887 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" event={"ID":"0f191422-4eb5-4b03-9163-3492c06004a4","Type":"ContainerDied","Data":"4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14"} Oct 31 13:20:09.830045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.829947 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh" event={"ID":"0f191422-4eb5-4b03-9163-3492c06004a4","Type":"ContainerDied","Data":"fe51dc23ad2d68a037d84a71becb1f98e6f773cf4dd40c0360d65405b9ef5210"} Oct 31 13:20:09.830045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.829980 5074 scope.go:122] "RemoveContainer" containerID="4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14" Oct 31 13:20:09.833455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.833340 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" event={"ID":"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0","Type":"ContainerStarted","Data":"df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b"} Oct 31 13:20:09.833455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.833411 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" event={"ID":"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0","Type":"ContainerStarted","Data":"be0442fcc3ed93197f51bbc002c7402cfa6919caae54f5fe613f1cfd1169ab8f"} Oct 31 13:20:09.834186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.834154 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:09.838145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.838113 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" event={"ID":"a57421ac-c611-4bee-a0fe-a7e4b143b00a","Type":"ContainerStarted","Data":"7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5"} Oct 31 13:20:09.838280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.838168 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" event={"ID":"a57421ac-c611-4bee-a0fe-a7e4b143b00a","Type":"ContainerStarted","Data":"61df2449b4b9fb89648cbd8adcf8906580cf55402a9409674437d3d59a4a8b91"} Oct 31 13:20:09.838280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.838215 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:09.844159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.844132 5074 scope.go:122] "RemoveContainer" containerID="4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14" Oct 31 13:20:09.844448 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:09.844425 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14\": container with ID starting with 4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14 not found: ID does not exist" containerID="4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14" Oct 31 13:20:09.844520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.844461 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14"} err="failed to get container status \"4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14\": rpc error: code = NotFound desc = could not find container \"4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14\": container with ID starting with 4c64dec8b7bdf0f5e05e00108d05d2d172ff029772e52b8d805daf5a0fe67d14 not found: ID does not exist" Oct 31 13:20:09.860595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.860543 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" podStartSLOduration=1.860529621 podStartE2EDuration="1.860529621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:09.855756787 +0000 UTC m=+2904.033474772" watchObservedRunningTime="2026-10-31 13:20:09.860529621 +0000 UTC m=+2904.038247638" Oct 31 13:20:09.879784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.879754 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh"] Oct 31 13:20:09.889576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.889527 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2ef686/payload-processing-external-model-e2e-extmod-a-2ef686-78875ghqh"] Oct 31 13:20:09.898064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.897999 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" podStartSLOduration=1.897973554 podStartE2EDuration="1.897973554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:09.893546011 +0000 UTC m=+2904.071264018" watchObservedRunningTime="2026-10-31 13:20:09.897973554 +0000 UTC m=+2904.075691539" Oct 31 13:20:09.942251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:09.942213 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f191422-4eb5-4b03-9163-3492c06004a4" path="/var/lib/kubelet/pods/0f191422-4eb5-4b03-9163-3492c06004a4/volumes" Oct 31 13:20:13.044755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.044705 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4"] Oct 31 13:20:13.048873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.048842 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0f191422-4eb5-4b03-9163-3492c06004a4" containerName="payload-processing" Oct 31 13:20:13.053809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.053722 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f191422-4eb5-4b03-9163-3492c06004a4" containerName="payload-processing" Oct 31 13:20:13.054205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.054182 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f191422-4eb5-4b03-9163-3492c06004a4" containerName="payload-processing" Oct 31 13:20:13.061653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.060109 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4"] Oct 31 13:20:13.061653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.060263 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.064971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.064187 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"agc-cred-b4bf09cb0482\"" Oct 31 13:20:13.065573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.065548 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"payload-processing-external-model-e2e--ef24cfc7-dockercfg-gw8dr\"" Oct 31 13:20:13.065779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.065759 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"routing-overlay\"" Oct 31 13:20:13.065906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.065888 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"openshift-service-ca.crt\"" Oct 31 13:20:13.065985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.065955 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"kube-root-ca.crt\"" Oct 31 13:20:13.066067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.066022 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5dc570\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5dc570\"" Oct 31 13:20:13.152668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.148835 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.152668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.148905 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.152668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.148973 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9cx2\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-kube-api-access-m9cx2\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.152668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.149009 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.462811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.454040 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.462811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.457681 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.474695 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:13.472254 5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5dc570/agc-cred-b4bf09cb0482: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:20:13.474695 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:13.472286 5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:20:13.474695 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:13.472359 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials podName:84bc2ba7-470d-4b41-b4c7-458b063ed2fc nodeName:}" failed. No retries permitted until 2026-10-31 13:20:13.972335237 +0000 UTC m=+2908.150053202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" (UID: "84bc2ba7-470d-4b41-b4c7-458b063ed2fc") : secret "agc-cred-b4bf09cb0482" not found Oct 31 13:20:13.515311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.515258 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.519766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.519735 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9cx2\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-kube-api-access-m9cx2\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.520420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.520392 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.535260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.535229 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.583476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.565732 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9cx2\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-kube-api-access-m9cx2\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:13.641833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.641795 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4"] Oct 31 13:20:13.643677 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:13.643620 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" podUID="84bc2ba7-470d-4b41-b4c7-458b063ed2fc" Oct 31 13:20:13.942654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:13.919162 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:14.017656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.013856 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:14.068713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.062721 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-kube-api-access-m9cx2\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-kube-api-access-m9cx2\") pod \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " Oct 31 13:20:14.068713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.062804 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-plugins-config-volume\") pod \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " Oct 31 13:20:14.068713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.062892 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-routing-overlay-volume\") pod \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " Oct 31 13:20:14.068713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.065288 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4\" (UID: \"84bc2ba7-470d-4b41-b4c7-458b063ed2fc\") " pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:14.091663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.072768 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-plugins-config-volume" pod "84bc2ba7-470d-4b41-b4c7-458b063ed2fc" (UID: "84bc2ba7-470d-4b41-b4c7-458b063ed2fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:14.091663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.073787 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-routing-overlay-volume" pod "84bc2ba7-470d-4b41-b4c7-458b063ed2fc" (UID: "84bc2ba7-470d-4b41-b4c7-458b063ed2fc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:14.091663 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:14.074502 5074 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5dc570/agc-cred-b4bf09cb0482: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:20:14.091663 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:14.074519 5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4: secret "agc-cred-b4bf09cb0482" not found Oct 31 13:20:14.091663 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:14.074621 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials podName:84bc2ba7-470d-4b41-b4c7-458b063ed2fc nodeName:}" failed. No retries permitted until 2026-10-31 13:20:15.074600465 +0000 UTC m=+2909.252318444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" (UID: "84bc2ba7-470d-4b41-b4c7-458b063ed2fc") : secret "agc-cred-b4bf09cb0482" not found Oct 31 13:20:14.091663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.082276 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:14.091663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.082311 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:14.138681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.120961 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-kube-api-access-m9cx2" pod "84bc2ba7-470d-4b41-b4c7-458b063ed2fc" (UID: "84bc2ba7-470d-4b41-b4c7-458b063ed2fc"). InnerVolumeSpecName "kube-api-access-m9cx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:14.183494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.183453 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9cx2\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-kube-api-access-m9cx2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:14.928271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.928230 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4" Oct 31 13:20:14.972297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.972257 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4"] Oct 31 13:20:14.980298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:14.980259 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5dc570/payload-processing-external-model-e2e-extmod-a-5dc570-7c8ff5vh4"] Oct 31 13:20:15.106004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:15.105958 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bc2ba7-470d-4b41-b4c7-458b063ed2fc-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:15.941899 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:15.941861 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bc2ba7-470d-4b41-b4c7-458b063ed2fc" path="/var/lib/kubelet/pods/84bc2ba7-470d-4b41-b4c7-458b063ed2fc/volumes" Oct 31 13:20:20.318870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.318830 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7f47568d6f-kd657"] Oct 31 13:20:20.322880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.322858 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.329139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.329112 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 13:20:20.329818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.329795 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-plugins-config-volume\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.329942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.329839 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-routing-overlay-volume\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.330016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.329935 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-provider-credentials\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.330097 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.330076 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlb2b\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-kube-api-access-dlb2b\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.330446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.330416 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 13:20:20.330446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.330437 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f42326e45f6c\"" Oct 31 13:20:20.330607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.330416 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 13:20:20.330734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.330709 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 13:20:20.344220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.344195 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f47568d6f-kd657"] Oct 31 13:20:20.473691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.469218 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-plugins-config-volume\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.473691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.469288 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-routing-overlay-volume\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.473691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.469363 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-provider-credentials\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.473691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.469472 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlb2b\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-kube-api-access-dlb2b\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.487655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.487165 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-plugins-config-volume\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.503879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.491206 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-routing-overlay-volume\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.514659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.509389 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-provider-credentials\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.544673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.533717 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlb2b\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-kube-api-access-dlb2b\") pod \"payload-processing-external-model-7f47568d6f-kd657\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.649715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.644927 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:20.867301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.859376 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:20:20.874945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:20.871849 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:21.025660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:21.025386 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8"] Oct 31 13:20:21.166785 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:20:21.166752 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2b180a_9b51_460c_b058_2f6ef9ea36e8.slice/crio-240ae0154c075d19fced6aa928c2ab12b620a2d7ae6746724196d5f56d06535a WatchSource:0}: Error finding container 240ae0154c075d19fced6aa928c2ab12b620a2d7ae6746724196d5f56d06535a: Status 404 returned error can't find the container with id 240ae0154c075d19fced6aa928c2ab12b620a2d7ae6746724196d5f56d06535a Oct 31 13:20:21.167813 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:21.167788 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f47568d6f-kd657"] Oct 31 13:20:22.029242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:22.029202 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" event={"ID":"3e2b180a-9b51-460c-b058-2f6ef9ea36e8","Type":"ContainerStarted","Data":"7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258"} Oct 31 13:20:22.029732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:22.029273 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" event={"ID":"3e2b180a-9b51-460c-b058-2f6ef9ea36e8","Type":"ContainerStarted","Data":"240ae0154c075d19fced6aa928c2ab12b620a2d7ae6746724196d5f56d06535a"} Oct 31 13:20:22.029732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:22.029306 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:22.050086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:22.050039 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" podStartSLOduration=2.050026907 podStartE2EDuration="2.050026907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:22.048863929 +0000 UTC m=+2916.226581916" watchObservedRunningTime="2026-10-31 13:20:22.050026907 +0000 UTC m=+2916.227744891" Oct 31 13:20:25.992568 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:25.992517 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:20:26.026321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.026230 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" podUID="c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" containerName="payload-processing" containerID="cri-o://df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b" gracePeriod=25 Oct 31 13:20:26.356901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.356874 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:26.413151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.413117 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-provider-credentials\") pod \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " Oct 31 13:20:26.413410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.413308 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-plugins-config-volume\") pod \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " Oct 31 13:20:26.413488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.413463 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-kube-api-access-s2gx6\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-kube-api-access-s2gx6\") pod \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " Oct 31 13:20:26.413552 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.413513 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-routing-overlay-volume\") pod \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\" (UID: \"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0\") " Oct 31 13:20:26.413956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.413924 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-plugins-config-volume" pod "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" (UID: "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:26.414250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.414227 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-routing-overlay-volume" pod "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" (UID: "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:26.414524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.414490 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:26.414604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.414531 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:26.417217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.417186 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-provider-credentials" pod "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" (UID: "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:26.417336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.417264 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-kube-api-access-s2gx6" pod "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" (UID: "c29c1e4d-8aa2-4be3-b773-b12b547aa0a0"). InnerVolumeSpecName "kube-api-access-s2gx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:26.515804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.515767 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gx6\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-kube-api-access-s2gx6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:26.515804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:26.515797 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:27.115600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.115557 5074 generic.go:356] "Generic (PLEG): container finished" podID="c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" containerID="df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b" exitCode=0 Oct 31 13:20:27.116200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.115625 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" event={"ID":"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0","Type":"ContainerDied","Data":"df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b"} Oct 31 13:20:27.116200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.115682 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" Oct 31 13:20:27.116200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.115702 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8" event={"ID":"c29c1e4d-8aa2-4be3-b773-b12b547aa0a0","Type":"ContainerDied","Data":"be0442fcc3ed93197f51bbc002c7402cfa6919caae54f5fe613f1cfd1169ab8f"} Oct 31 13:20:27.116200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.115736 5074 scope.go:122] "RemoveContainer" containerID="df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b" Oct 31 13:20:27.130132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.130108 5074 scope.go:122] "RemoveContainer" containerID="df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b" Oct 31 13:20:27.130445 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:27.130417 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b\": container with ID starting with df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b not found: ID does not exist" containerID="df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b" Oct 31 13:20:27.130519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.130454 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b"} err="failed to get container status \"df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b\": rpc error: code = NotFound desc = could not find container \"df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b\": container with ID starting with df700829fecb82b61573e5673d31e356696acf3c1db319c4daa2fe70c4bd080b not found: ID does not exist" Oct 31 13:20:27.155835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.155807 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8"] Oct 31 13:20:27.160200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.160177 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-7f777zfh8"] Oct 31 13:20:27.942679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:27.942599 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" path="/var/lib/kubelet/pods/c29c1e4d-8aa2-4be3-b773-b12b547aa0a0/volumes" Oct 31 13:20:30.981655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:30.980672 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr"] Oct 31 13:20:30.982158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:30.981812 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" containerName="payload-processing" Oct 31 13:20:30.982158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:30.981836 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" containerName="payload-processing" Oct 31 13:20:30.982158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:30.982084 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29c1e4d-8aa2-4be3-b773-b12b547aa0a0" containerName="payload-processing" Oct 31 13:20:30.988438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:30.988406 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.021454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.021416 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr"] Oct 31 13:20:31.090479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.067773 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2ccf99ff0d3b\"" Oct 31 13:20:31.090479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.090256 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-plugins-config-volume\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.090830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.090587 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-routing-overlay-volume\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.090830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.090770 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4rm\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-kube-api-access-7n4rm\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.095069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.094997 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-provider-credentials\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.097523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.097386 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-99ph4\"" Oct 31 13:20:31.196592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.196554 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-plugins-config-volume\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.196789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.196695 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-routing-overlay-volume\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.196789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.196783 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4rm\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-kube-api-access-7n4rm\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.196924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.196821 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-provider-credentials\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.197537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.197508 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-routing-overlay-volume\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.197702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.197543 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-plugins-config-volume\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.199489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.199465 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-provider-credentials\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.210103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.210078 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4rm\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-kube-api-access-7n4rm\") pod \"payload-processing-external-model-856786ccdc-bmdwr\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.462749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.462672 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:31.762210 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:20:31.762170 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39580112_a39f_4efb_beb7_37f00531978e.slice/crio-6a21bd2985c837d6ad3cff5c2880bcdd0df3d3161f618159ad9c4e27008545fd WatchSource:0}: Error finding container 6a21bd2985c837d6ad3cff5c2880bcdd0df3d3161f618159ad9c4e27008545fd: Status 404 returned error can't find the container with id 6a21bd2985c837d6ad3cff5c2880bcdd0df3d3161f618159ad9c4e27008545fd Oct 31 13:20:31.762956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:31.762931 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr"] Oct 31 13:20:32.211341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:32.211250 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" event={"ID":"39580112-a39f-4efb-beb7-37f00531978e","Type":"ContainerStarted","Data":"47e6ff853593fc30309d1baee66cf6ef9d95ed46197424db4db4a862e7f10f81"} Oct 31 13:20:32.211341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:32.211317 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" event={"ID":"39580112-a39f-4efb-beb7-37f00531978e","Type":"ContainerStarted","Data":"6a21bd2985c837d6ad3cff5c2880bcdd0df3d3161f618159ad9c4e27008545fd"} Oct 31 13:20:32.211858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:32.211510 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:32.241272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:32.241221 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" podStartSLOduration=2.24120681 podStartE2EDuration="2.24120681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:32.236256836 +0000 UTC m=+2926.413974820" watchObservedRunningTime="2026-10-31 13:20:32.24120681 +0000 UTC m=+2926.418924806" Oct 31 13:20:33.042487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:33.042453 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:43.224742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:43.224710 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:20:43.302190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:43.302154 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f47568d6f-kd657"] Oct 31 13:20:45.373096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.373054 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r"] Oct 31 13:20:45.377781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.377757 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.382673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.382625 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:20:45.383857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.383836 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:20:45.383971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.383873 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:20:45.384329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.383964 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:20:45.394891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.394864 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r"] Oct 31 13:20:45.409840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.409804 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.410010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.409955 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zvf\" (UniqueName: \"kubernetes.io/projected/11c67e78-4fd5-4241-836a-ccabf8cd3c96-kube-api-access-l6zvf\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.410010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.410002 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.511704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.511665 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.511918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.511791 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zvf\" (UniqueName: \"kubernetes.io/projected/11c67e78-4fd5-4241-836a-ccabf8cd3c96-kube-api-access-l6zvf\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.511918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.511819 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.512564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.512529 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.514345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.514317 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.524760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.524730 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zvf\" (UniqueName: \"kubernetes.io/projected/11c67e78-4fd5-4241-836a-ccabf8cd3c96-kube-api-access-l6zvf\") pod \"maas-api-revoke-sub-5dd25297-jc92r\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:45.696352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:45.696265 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:46.031052 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:20:46.031013 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c67e78_4fd5_4241_836a_ccabf8cd3c96.slice/crio-3839ec74b30104bcaf2c21572f8bfce195b6fe927c66d2120e37686b961ffa20 WatchSource:0}: Error finding container 3839ec74b30104bcaf2c21572f8bfce195b6fe927c66d2120e37686b961ffa20: Status 404 returned error can't find the container with id 3839ec74b30104bcaf2c21572f8bfce195b6fe927c66d2120e37686b961ffa20 Oct 31 13:20:46.031221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:46.031178 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r"] Oct 31 13:20:46.380456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:46.380419 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" event={"ID":"11c67e78-4fd5-4241-836a-ccabf8cd3c96","Type":"ContainerStarted","Data":"c444ee929bd025119f976f5f452b56388f64bab934518cdd8881ea8345cff9ad"} Oct 31 13:20:46.380957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:46.380485 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" event={"ID":"11c67e78-4fd5-4241-836a-ccabf8cd3c96","Type":"ContainerStarted","Data":"3839ec74b30104bcaf2c21572f8bfce195b6fe927c66d2120e37686b961ffa20"} Oct 31 13:20:47.390297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:47.390260 5074 generic.go:356] "Generic (PLEG): container finished" podID="11c67e78-4fd5-4241-836a-ccabf8cd3c96" containerID="c444ee929bd025119f976f5f452b56388f64bab934518cdd8881ea8345cff9ad" exitCode=0 Oct 31 13:20:47.390297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:47.390299 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" event={"ID":"11c67e78-4fd5-4241-836a-ccabf8cd3c96","Type":"ContainerDied","Data":"c444ee929bd025119f976f5f452b56388f64bab934518cdd8881ea8345cff9ad"} Oct 31 13:20:48.303054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.303010 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" podUID="3e2b180a-9b51-460c-b058-2f6ef9ea36e8" containerName="payload-processing" containerID="cri-o://7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258" gracePeriod=25 Oct 31 13:20:48.684705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.684665 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:48.696517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.696482 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-openshift-ca\") pod \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " Oct 31 13:20:48.696942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.696850 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c67e78-4fd5-4241-836a-ccabf8cd3c96-kube-api-access-l6zvf\" (UniqueName: \"kubernetes.io/projected/11c67e78-4fd5-4241-836a-ccabf8cd3c96-kube-api-access-l6zvf\") pod \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " Oct 31 13:20:48.696942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.696895 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-openshift-ca" pod "11c67e78-4fd5-4241-836a-ccabf8cd3c96" (UID: "11c67e78-4fd5-4241-836a-ccabf8cd3c96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:48.697123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.697082 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-xks-ca\") pod \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\" (UID: \"11c67e78-4fd5-4241-836a-ccabf8cd3c96\") " Oct 31 13:20:48.697837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.697802 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:48.699706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.699679 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-xks-ca" pod "11c67e78-4fd5-4241-836a-ccabf8cd3c96" (UID: "11c67e78-4fd5-4241-836a-ccabf8cd3c96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:20:48.703353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.703318 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c67e78-4fd5-4241-836a-ccabf8cd3c96-kube-api-access-l6zvf" pod "11c67e78-4fd5-4241-836a-ccabf8cd3c96" (UID: "11c67e78-4fd5-4241-836a-ccabf8cd3c96"). InnerVolumeSpecName "kube-api-access-l6zvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:48.742592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.742567 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:48.798721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.798689 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-plugins-config-volume\") pod \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " Oct 31 13:20:48.798921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.798804 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-provider-credentials\") pod \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " Oct 31 13:20:48.799004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.798951 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-kube-api-access-dlb2b\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-kube-api-access-dlb2b\") pod \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " Oct 31 13:20:48.799077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.799054 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-plugins-config-volume" pod "3e2b180a-9b51-460c-b058-2f6ef9ea36e8" (UID: "3e2b180a-9b51-460c-b058-2f6ef9ea36e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:48.799183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.799167 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-routing-overlay-volume\") pod \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\" (UID: \"3e2b180a-9b51-460c-b058-2f6ef9ea36e8\") " Oct 31 13:20:48.799621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.799578 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-routing-overlay-volume" pod "3e2b180a-9b51-460c-b058-2f6ef9ea36e8" (UID: "3e2b180a-9b51-460c-b058-2f6ef9ea36e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:20:48.800069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.800021 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zvf\" (UniqueName: \"kubernetes.io/projected/11c67e78-4fd5-4241-836a-ccabf8cd3c96-kube-api-access-l6zvf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:48.800069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.800051 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:48.800069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.800067 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c67e78-4fd5-4241-836a-ccabf8cd3c96-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:48.800299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.800081 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:48.801409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.801378 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-provider-credentials" pod "3e2b180a-9b51-460c-b058-2f6ef9ea36e8" (UID: "3e2b180a-9b51-460c-b058-2f6ef9ea36e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:48.801510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.801437 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-kube-api-access-dlb2b" pod "3e2b180a-9b51-460c-b058-2f6ef9ea36e8" (UID: "3e2b180a-9b51-460c-b058-2f6ef9ea36e8"). InnerVolumeSpecName "kube-api-access-dlb2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:20:48.901917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.901829 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlb2b\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-kube-api-access-dlb2b\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:48.901917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:48.901860 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e2b180a-9b51-460c-b058-2f6ef9ea36e8-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:20:49.412222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.412189 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" Oct 31 13:20:49.412425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.412216 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r" event={"ID":"11c67e78-4fd5-4241-836a-ccabf8cd3c96","Type":"ContainerDied","Data":"3839ec74b30104bcaf2c21572f8bfce195b6fe927c66d2120e37686b961ffa20"} Oct 31 13:20:49.412425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.412274 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3839ec74b30104bcaf2c21572f8bfce195b6fe927c66d2120e37686b961ffa20" Oct 31 13:20:49.413852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.413824 5074 generic.go:356] "Generic (PLEG): container finished" podID="3e2b180a-9b51-460c-b058-2f6ef9ea36e8" containerID="7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258" exitCode=0 Oct 31 13:20:49.413989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.413864 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" Oct 31 13:20:49.413989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.413897 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" event={"ID":"3e2b180a-9b51-460c-b058-2f6ef9ea36e8","Type":"ContainerDied","Data":"7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258"} Oct 31 13:20:49.413989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.413946 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f47568d6f-kd657" event={"ID":"3e2b180a-9b51-460c-b058-2f6ef9ea36e8","Type":"ContainerDied","Data":"240ae0154c075d19fced6aa928c2ab12b620a2d7ae6746724196d5f56d06535a"} Oct 31 13:20:49.413989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.413974 5074 scope.go:122] "RemoveContainer" containerID="7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258" Oct 31 13:20:49.427415 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.427395 5074 scope.go:122] "RemoveContainer" containerID="7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258" Oct 31 13:20:49.427800 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:49.427778 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258\": container with ID starting with 7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258 not found: ID does not exist" containerID="7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258" Oct 31 13:20:49.427907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.427811 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258"} err="failed to get container status \"7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258\": rpc error: code = NotFound desc = could not find container \"7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258\": container with ID starting with 7f0a35d215cddeccd3e6aff9082c907d5ee647f607a573239d318693601a0258 not found: ID does not exist" Oct 31 13:20:49.449934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.449908 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f47568d6f-kd657"] Oct 31 13:20:49.457371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.457343 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f47568d6f-kd657"] Oct 31 13:20:49.943311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:49.943260 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e2b180a-9b51-460c-b058-2f6ef9ea36e8" path="/var/lib/kubelet/pods/3e2b180a-9b51-460c-b058-2f6ef9ea36e8/volumes" Oct 31 13:20:55.984326 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:20:55.984285 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:20:58.588070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.587211 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-hcghl"] Oct 31 13:20:58.591682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.591425 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3e2b180a-9b51-460c-b058-2f6ef9ea36e8" containerName="payload-processing" Oct 31 13:20:58.591682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.591472 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2b180a-9b51-460c-b058-2f6ef9ea36e8" containerName="payload-processing" Oct 31 13:20:58.591682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.591524 5074 cpu_manager.go:414] "containerMap: removing container" podUID="11c67e78-4fd5-4241-836a-ccabf8cd3c96" containerName="revoke-keys" Oct 31 13:20:58.591682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.591532 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c67e78-4fd5-4241-836a-ccabf8cd3c96" containerName="revoke-keys" Oct 31 13:20:58.592017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.591845 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c67e78-4fd5-4241-836a-ccabf8cd3c96" containerName="revoke-keys" Oct 31 13:20:58.592017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.591878 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e2b180a-9b51-460c-b058-2f6ef9ea36e8" containerName="payload-processing" Oct 31 13:20:58.601243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.601190 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-hcghl"] Oct 31 13:20:58.601428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.601343 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.607192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.607162 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3ee368d0531c\"" Oct 31 13:20:58.714533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.714482 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-routing-overlay-volume\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.714780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.714602 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62vsw\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-kube-api-access-62vsw\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.714927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.714894 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-plugins-config-volume\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.715027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.714996 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-provider-credentials\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.817046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.817007 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-plugins-config-volume\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.817258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.817056 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-provider-credentials\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.817258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.817204 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-routing-overlay-volume\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.817398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.817257 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vsw\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-kube-api-access-62vsw\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.817891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.817860 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-routing-overlay-volume\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.818114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.818086 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-plugins-config-volume\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.819960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.819934 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-provider-credentials\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.829785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.829756 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62vsw\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-kube-api-access-62vsw\") pod \"payload-processing-external-model-655bb989fc-hcghl\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:58.926169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:58.925908 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:20:59.653259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:59.653218 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-hcghl"] Oct 31 13:20:59.658096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:59.658011 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-hcghl"] Oct 31 13:20:59.675643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:20:59.674207 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" event={"ID":"b8ecc3ba-5933-44d1-89e3-cc25a0b219be","Type":"ContainerStarted","Data":"32acb78fa6b98d1cde32b3a365a89bbb5b7dc6a5710aa19dd19ebe9424f5656c"} Oct 31 13:21:00.692009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:00.691967 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" event={"ID":"b8ecc3ba-5933-44d1-89e3-cc25a0b219be","Type":"ContainerStarted","Data":"eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c"} Oct 31 13:21:00.692402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:00.692157 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:21:00.742578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:00.742521 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" podStartSLOduration=2.742507248 podStartE2EDuration="2.742507248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:00.737510574 +0000 UTC m=+2954.915228560" watchObservedRunningTime="2026-10-31 13:21:00.742507248 +0000 UTC m=+2954.920225260" Oct 31 13:21:01.233396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:01.233358 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:21:05.693077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:05.693035 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" podUID="b8ecc3ba-5933-44d1-89e3-cc25a0b219be" containerName="payload-processing" containerID="cri-o://eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c" gracePeriod=25 Oct 31 13:21:06.074722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.074684 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:21:06.175933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.175895 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-routing-overlay-volume\") pod \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " Oct 31 13:21:06.176142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.175997 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-provider-credentials\") pod \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " Oct 31 13:21:06.176142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.176080 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-kube-api-access-62vsw\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-kube-api-access-62vsw\") pod \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " Oct 31 13:21:06.176361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.176324 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-routing-overlay-volume" pod "b8ecc3ba-5933-44d1-89e3-cc25a0b219be" (UID: "b8ecc3ba-5933-44d1-89e3-cc25a0b219be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:06.176361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.176342 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-plugins-config-volume\") pod \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\" (UID: \"b8ecc3ba-5933-44d1-89e3-cc25a0b219be\") " Oct 31 13:21:06.176766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.176739 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-plugins-config-volume" pod "b8ecc3ba-5933-44d1-89e3-cc25a0b219be" (UID: "b8ecc3ba-5933-44d1-89e3-cc25a0b219be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:06.176977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.176960 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:06.176977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.176978 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:06.178452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.178426 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-kube-api-access-62vsw" pod "b8ecc3ba-5933-44d1-89e3-cc25a0b219be" (UID: "b8ecc3ba-5933-44d1-89e3-cc25a0b219be"). InnerVolumeSpecName "kube-api-access-62vsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:06.178618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.178599 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-provider-credentials" pod "b8ecc3ba-5933-44d1-89e3-cc25a0b219be" (UID: "b8ecc3ba-5933-44d1-89e3-cc25a0b219be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:06.278825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.278772 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:06.278825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.278816 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62vsw\" (UniqueName: \"kubernetes.io/projected/b8ecc3ba-5933-44d1-89e3-cc25a0b219be-kube-api-access-62vsw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:06.784655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.776495 5074 generic.go:356] "Generic (PLEG): container finished" podID="b8ecc3ba-5933-44d1-89e3-cc25a0b219be" containerID="eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c" exitCode=0 Oct 31 13:21:06.784655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.776703 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" event={"ID":"b8ecc3ba-5933-44d1-89e3-cc25a0b219be","Type":"ContainerDied","Data":"eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c"} Oct 31 13:21:06.784655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.776757 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" event={"ID":"b8ecc3ba-5933-44d1-89e3-cc25a0b219be","Type":"ContainerDied","Data":"32acb78fa6b98d1cde32b3a365a89bbb5b7dc6a5710aa19dd19ebe9424f5656c"} Oct 31 13:21:06.784655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.776789 5074 scope.go:122] "RemoveContainer" containerID="eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c" Oct 31 13:21:06.784655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.776949 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655bb989fc-hcghl" Oct 31 13:21:06.812961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.812931 5074 scope.go:122] "RemoveContainer" containerID="eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c" Oct 31 13:21:06.813526 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:06.813387 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c\": container with ID starting with eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c not found: ID does not exist" containerID="eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c" Oct 31 13:21:06.813526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.813433 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c"} err="failed to get container status \"eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c\": rpc error: code = NotFound desc = could not find container \"eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c\": container with ID starting with eb454e008276b235e37945145c9c9d5803048bee2de3c00c6649aac7395d314c not found: ID does not exist" Oct 31 13:21:06.819670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.819596 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-hcghl"] Oct 31 13:21:06.827476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:06.827442 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-655bb989fc-hcghl"] Oct 31 13:21:07.941278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:07.941234 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ecc3ba-5933-44d1-89e3-cc25a0b219be" path="/var/lib/kubelet/pods/b8ecc3ba-5933-44d1-89e3-cc25a0b219be/volumes" Oct 31 13:21:08.240788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.240750 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm"] Oct 31 13:21:08.241460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.241446 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b8ecc3ba-5933-44d1-89e3-cc25a0b219be" containerName="payload-processing" Oct 31 13:21:08.241513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.241461 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ecc3ba-5933-44d1-89e3-cc25a0b219be" containerName="payload-processing" Oct 31 13:21:08.241618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.241609 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ecc3ba-5933-44d1-89e3-cc25a0b219be" containerName="payload-processing" Oct 31 13:21:08.247068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.247044 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.251881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.251713 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:21:08.252513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.252490 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:21:08.252703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.252609 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:21:08.252703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.252683 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:21:08.257695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.257509 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm"] Oct 31 13:21:08.340003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.339966 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.340212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.340159 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.340299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.340209 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgpsz\" (UniqueName: \"kubernetes.io/projected/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-kube-api-access-cgpsz\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.441550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.441515 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.441817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.441687 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.441817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.441750 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgpsz\" (UniqueName: \"kubernetes.io/projected/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-kube-api-access-cgpsz\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.442328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.442308 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.444124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.444104 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.455228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.455196 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgpsz\" (UniqueName: \"kubernetes.io/projected/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-kube-api-access-cgpsz\") pod \"maas-api-revoke-sub-c20705af-529sm\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.575596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.575502 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:08.884864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:08.884831 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm"] Oct 31 13:21:08.889042 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:21:08.889012 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b022f1_d4f7_47a8_b086_3b8c53b85ebb.slice/crio-9e3b1e889c93d9bde810dcb3c1fb4e31d498532a8e88a904b2d928f7bbc14235 WatchSource:0}: Error finding container 9e3b1e889c93d9bde810dcb3c1fb4e31d498532a8e88a904b2d928f7bbc14235: Status 404 returned error can't find the container with id 9e3b1e889c93d9bde810dcb3c1fb4e31d498532a8e88a904b2d928f7bbc14235 Oct 31 13:21:09.404787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.404741 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh"] Oct 31 13:21:09.411935 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.411909 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.438547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.438516 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c53c6c7bff0d\"" Oct 31 13:21:09.457434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.457396 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh"] Oct 31 13:21:09.578755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.578711 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkm9q\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-kube-api-access-rkm9q\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.578944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.578772 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-plugins-config-volume\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.579027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.578999 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.579190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.579171 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-routing-overlay-volume\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.680300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.680261 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkm9q\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-kube-api-access-rkm9q\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.680499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.680310 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-plugins-config-volume\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.680499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.680386 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.680499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.680470 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-routing-overlay-volume\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.681274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.681250 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-routing-overlay-volume\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.681405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.681274 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-plugins-config-volume\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.683520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.683493 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.695564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.695534 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkm9q\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-kube-api-access-rkm9q\") pod \"payload-processing-external-model-6bbd96f97b-2rhjh\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.728293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.728258 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:09.819449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.819375 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" event={"ID":"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb","Type":"ContainerStarted","Data":"5f7e699d940046b99caed3055f42784fb5e760264496909dcaf1d695d0a09255"} Oct 31 13:21:09.819661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.819471 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" event={"ID":"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb","Type":"ContainerStarted","Data":"9e3b1e889c93d9bde810dcb3c1fb4e31d498532a8e88a904b2d928f7bbc14235"} Oct 31 13:21:09.841785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.841725 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" podStartSLOduration=1.8417096210000001 podStartE2EDuration="1.841709621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:09.838793802 +0000 UTC m=+2964.016511786" watchObservedRunningTime="2026-10-31 13:21:09.841709621 +0000 UTC m=+2964.019427604" Oct 31 13:21:09.960205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.960158 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj"] Oct 31 13:21:09.975457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.975425 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:09.990334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:09.989959 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj"] Oct 31 13:21:10.050496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.043959 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh"] Oct 31 13:21:10.089578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.089547 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.090301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.089908 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.090301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.090136 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfxp\" (UniqueName: \"kubernetes.io/projected/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-kube-api-access-pjfxp\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.192061 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.192030 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.192249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.192215 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfxp\" (UniqueName: \"kubernetes.io/projected/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-kube-api-access-pjfxp\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.192332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.192301 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.193099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.193078 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.194958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.194905 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.205078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.205049 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfxp\" (UniqueName: \"kubernetes.io/projected/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-kube-api-access-pjfxp\") pod \"maas-api-revoke-sub-d86e5c27-ls8jj\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.267579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.267479 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh"] Oct 31 13:21:10.323659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.321145 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:10.829113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.829073 5074 generic.go:356] "Generic (PLEG): container finished" podID="f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" containerID="5f7e699d940046b99caed3055f42784fb5e760264496909dcaf1d695d0a09255" exitCode=0 Oct 31 13:21:10.829537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.829155 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" event={"ID":"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb","Type":"ContainerDied","Data":"5f7e699d940046b99caed3055f42784fb5e760264496909dcaf1d695d0a09255"} Oct 31 13:21:10.831145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.831118 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" event={"ID":"a1b8b848-0910-4a49-b1c6-9734c3a6cc44","Type":"ContainerStarted","Data":"2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806"} Oct 31 13:21:10.831305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.831163 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" event={"ID":"a1b8b848-0910-4a49-b1c6-9734c3a6cc44","Type":"ContainerStarted","Data":"8e4931aa1bf91abf912eb20df11509e294823cf425671483fe442be75fd5e33a"} Oct 31 13:21:10.831305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.831201 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:10.926279 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:21:10.926233 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4ab14a_fdcd_4f0a_afcf_60b51756dfdf.slice/crio-fc8463b03d87c685716b813955a5bd223ebad9b04cd5d44d063256931170258d WatchSource:0}: Error finding container fc8463b03d87c685716b813955a5bd223ebad9b04cd5d44d063256931170258d: Status 404 returned error can't find the container with id fc8463b03d87c685716b813955a5bd223ebad9b04cd5d44d063256931170258d Oct 31 13:21:10.929127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.929101 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj"] Oct 31 13:21:10.937755 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:10.936868 5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c53c6c7bff0d: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:10.937755 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:10.936897 5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:10.937755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:10.937529 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" podStartSLOduration=1.937511798 podStartE2EDuration="1.937511798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:10.933040142 +0000 UTC m=+2965.110758127" watchObservedRunningTime="2026-10-31 13:21:10.937511798 +0000 UTC m=+2965.115229785" Oct 31 13:21:10.938107 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:10.938086 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials podName:a1b8b848-0910-4a49-b1c6-9734c3a6cc44 nodeName:}" failed. No retries permitted until 2026-10-31 13:21:11.438058279 +0000 UTC m=+2965.615776249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials") pod "payload-processing-external-model-6bbd96f97b-2rhjh" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44") : secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:11.459653 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:11.457601 5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c53c6c7bff0d: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:11.459653 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:11.457659 5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:11.459653 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:11.457722 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials podName:a1b8b848-0910-4a49-b1c6-9734c3a6cc44 nodeName:}" failed. No retries permitted until 2026-10-31 13:21:12.457700755 +0000 UTC m=+2966.635418732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials") pod "payload-processing-external-model-6bbd96f97b-2rhjh" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44") : secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:11.844236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:11.844196 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" event={"ID":"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf","Type":"ContainerStarted","Data":"295d70da1c9beb496770bf82ec187231e75c4f4b92695a868c889bf06f76f088"} Oct 31 13:21:11.844829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:11.844255 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" event={"ID":"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf","Type":"ContainerStarted","Data":"fc8463b03d87c685716b813955a5bd223ebad9b04cd5d44d063256931170258d"} Oct 31 13:21:11.866089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:11.866032 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" podStartSLOduration=2.866001353 podStartE2EDuration="2.866001353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:11.865026895 +0000 UTC m=+2966.042744882" watchObservedRunningTime="2026-10-31 13:21:11.866001353 +0000 UTC m=+2966.043719315" Oct 31 13:21:12.045735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.045706 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:12.169428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.169337 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-xks-ca\") pod \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " Oct 31 13:21:12.169760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.169727 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-openshift-ca\") pod \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " Oct 31 13:21:12.170108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.170083 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-kube-api-access-cgpsz\" (UniqueName: \"kubernetes.io/projected/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-kube-api-access-cgpsz\") pod \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\" (UID: \"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb\") " Oct 31 13:21:12.170522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.170291 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-openshift-ca" pod "f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" (UID: "f2b022f1-d4f7-47a8-b086-3b8c53b85ebb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:12.171148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.171120 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:12.172459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.172429 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-xks-ca" pod "f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" (UID: "f2b022f1-d4f7-47a8-b086-3b8c53b85ebb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:21:12.172581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.172531 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-kube-api-access-cgpsz" pod "f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" (UID: "f2b022f1-d4f7-47a8-b086-3b8c53b85ebb"). InnerVolumeSpecName "kube-api-access-cgpsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:12.272356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.272319 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgpsz\" (UniqueName: \"kubernetes.io/projected/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-kube-api-access-cgpsz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:12.272356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.272356 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:12.480255 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:12.480216 5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c53c6c7bff0d: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:12.480255 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:12.480252 5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:12.480485 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:12.480326 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials podName:a1b8b848-0910-4a49-b1c6-9734c3a6cc44 nodeName:}" failed. No retries permitted until 2026-10-31 13:21:14.480306316 +0000 UTC m=+2968.658024293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials") pod "payload-processing-external-model-6bbd96f97b-2rhjh" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44") : secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:12.854302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.854210 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" Oct 31 13:21:12.854302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.854241 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm" event={"ID":"f2b022f1-d4f7-47a8-b086-3b8c53b85ebb","Type":"ContainerDied","Data":"9e3b1e889c93d9bde810dcb3c1fb4e31d498532a8e88a904b2d928f7bbc14235"} Oct 31 13:21:12.854829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.854330 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e3b1e889c93d9bde810dcb3c1fb4e31d498532a8e88a904b2d928f7bbc14235" Oct 31 13:21:12.856184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.856155 5074 generic.go:356] "Generic (PLEG): container finished" podID="9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" containerID="295d70da1c9beb496770bf82ec187231e75c4f4b92695a868c889bf06f76f088" exitCode=0 Oct 31 13:21:12.856335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:12.856207 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" event={"ID":"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf","Type":"ContainerDied","Data":"295d70da1c9beb496770bf82ec187231e75c4f4b92695a868c889bf06f76f088"} Oct 31 13:21:14.049699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.049672 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:14.106328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.106291 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-kube-api-access-pjfxp\" (UniqueName: \"kubernetes.io/projected/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-kube-api-access-pjfxp\") pod \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " Oct 31 13:21:14.107867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.106386 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-openshift-ca\") pod \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " Oct 31 13:21:14.107867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.106485 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-xks-ca\") pod \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\" (UID: \"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf\") " Oct 31 13:21:14.108546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.108514 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-openshift-ca" pod "9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" (UID: "9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:14.110177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.110148 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-xks-ca" pod "9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" (UID: "9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:21:14.110588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.110558 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-kube-api-access-pjfxp" pod "9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" (UID: "9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf"). InnerVolumeSpecName "kube-api-access-pjfxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:14.209969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.209928 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:14.209969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.209972 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjfxp\" (UniqueName: \"kubernetes.io/projected/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-kube-api-access-pjfxp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:14.210242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.209988 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:14.524747 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:14.524710 5074 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c53c6c7bff0d: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:14.525011 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:14.524992 5074 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh: secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:14.525172 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:14.525160 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials podName:a1b8b848-0910-4a49-b1c6-9734c3a6cc44 nodeName:}" failed. No retries permitted until 2026-10-31 13:21:18.525136426 +0000 UTC m=+2972.702854402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials") pod "payload-processing-external-model-6bbd96f97b-2rhjh" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44") : secret "agc-cred-c53c6c7bff0d" not found Oct 31 13:21:14.882697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.882587 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" event={"ID":"9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf","Type":"ContainerDied","Data":"fc8463b03d87c685716b813955a5bd223ebad9b04cd5d44d063256931170258d"} Oct 31 13:21:14.882998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.882972 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8463b03d87c685716b813955a5bd223ebad9b04cd5d44d063256931170258d" Oct 31 13:21:14.883170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:14.882746 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj" Oct 31 13:21:16.845491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:16.845422 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" podUID="a1b8b848-0910-4a49-b1c6-9734c3a6cc44" containerName="payload-processing" containerID="cri-o://2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806" gracePeriod=25 Oct 31 13:21:17.275266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.275233 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:17.311458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.311418 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-plugins-config-volume\") pod \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " Oct 31 13:21:17.311694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.311561 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials\") pod \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " Oct 31 13:21:17.311694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.311584 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-routing-overlay-volume\") pod \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " Oct 31 13:21:17.311694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.311691 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-kube-api-access-rkm9q\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-kube-api-access-rkm9q\") pod \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\" (UID: \"a1b8b848-0910-4a49-b1c6-9734c3a6cc44\") " Oct 31 13:21:17.312108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.312072 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-plugins-config-volume" pod "a1b8b848-0910-4a49-b1c6-9734c3a6cc44" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:17.312886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.312186 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-routing-overlay-volume" pod "a1b8b848-0910-4a49-b1c6-9734c3a6cc44" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:17.312886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.312305 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:17.312886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.312320 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:17.314905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.314879 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials" pod "a1b8b848-0910-4a49-b1c6-9734c3a6cc44" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:17.316799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.316776 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-kube-api-access-rkm9q" pod "a1b8b848-0910-4a49-b1c6-9734c3a6cc44" (UID: "a1b8b848-0910-4a49-b1c6-9734c3a6cc44"). InnerVolumeSpecName "kube-api-access-rkm9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:17.419397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.418904 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:17.419397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.418949 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkm9q\" (UniqueName: \"kubernetes.io/projected/a1b8b848-0910-4a49-b1c6-9734c3a6cc44-kube-api-access-rkm9q\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:17.918147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.918105 5074 generic.go:356] "Generic (PLEG): container finished" podID="a1b8b848-0910-4a49-b1c6-9734c3a6cc44" containerID="2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806" exitCode=0 Oct 31 13:21:17.918564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.918156 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" Oct 31 13:21:17.918564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.918196 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" event={"ID":"a1b8b848-0910-4a49-b1c6-9734c3a6cc44","Type":"ContainerDied","Data":"2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806"} Oct 31 13:21:17.918564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.918299 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh" event={"ID":"a1b8b848-0910-4a49-b1c6-9734c3a6cc44","Type":"ContainerDied","Data":"8e4931aa1bf91abf912eb20df11509e294823cf425671483fe442be75fd5e33a"} Oct 31 13:21:17.918564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.918333 5074 scope.go:122] "RemoveContainer" containerID="2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806" Oct 31 13:21:17.931209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.931180 5074 scope.go:122] "RemoveContainer" containerID="2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806" Oct 31 13:21:17.931514 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:17.931495 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806\": container with ID starting with 2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806 not found: ID does not exist" containerID="2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806" Oct 31 13:21:17.931601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.931521 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806"} err="failed to get container status \"2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806\": rpc error: code = NotFound desc = could not find container \"2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806\": container with ID starting with 2ccc1cc4dbb6d3636325e8c8422037d0849df0f1916ef6bead24361760b03806 not found: ID does not exist" Oct 31 13:21:17.947510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.947485 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh"] Oct 31 13:21:17.952354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:17.952328 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bbd96f97b-2rhjh"] Oct 31 13:21:19.941795 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:19.941762 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b8b848-0910-4a49-b1c6-9734c3a6cc44" path="/var/lib/kubelet/pods/a1b8b848-0910-4a49-b1c6-9734c3a6cc44/volumes" Oct 31 13:21:21.221278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.221243 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9"] Oct 31 13:21:21.223602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.223571 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" containerName="revoke-keys" Oct 31 13:21:21.223833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.223813 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" containerName="revoke-keys" Oct 31 13:21:21.224099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.224086 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a1b8b848-0910-4a49-b1c6-9734c3a6cc44" containerName="payload-processing" Oct 31 13:21:21.224192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.224181 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b8b848-0910-4a49-b1c6-9734c3a6cc44" containerName="payload-processing" Oct 31 13:21:21.224293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.224283 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" containerName="revoke-keys" Oct 31 13:21:21.224369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.224359 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" containerName="revoke-keys" Oct 31 13:21:21.224898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.224878 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b8b848-0910-4a49-b1c6-9734c3a6cc44" containerName="payload-processing" Oct 31 13:21:21.225015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.225004 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" containerName="revoke-keys" Oct 31 13:21:21.225109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.225100 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" containerName="revoke-keys" Oct 31 13:21:21.262730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.262535 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9"] Oct 31 13:21:21.262923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.262746 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.268450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.267507 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-44a9af9b2905\"" Oct 31 13:21:21.268450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.267683 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-e2e--427d574a-dockercfg-xc26m\"" Oct 31 13:21:21.268450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.267855 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"agc-cred-93157c2e27cd\"" Oct 31 13:21:21.271729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.269520 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-54a16b\"" Oct 31 13:21:21.271729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.269616 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"routing-overlay\"" Oct 31 13:21:21.271729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.269784 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"openshift-service-ca.crt\"" Oct 31 13:21:21.271729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.269848 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-54a16b\"/\"kube-root-ca.crt\"" Oct 31 13:21:21.294372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.294341 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.294719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.294692 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.294923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.294907 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.295197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.295180 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8j8\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-kube-api-access-dv8j8\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.400655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.399931 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.400655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.400072 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8j8\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-kube-api-access-dv8j8\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.400655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.400246 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.400655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.400382 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.402563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.401561 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.402563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.402359 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.405819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.405795 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.417153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.417065 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8j8\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-kube-api-access-dv8j8\") pod \"payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.601713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.599921 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:21.601713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.600263 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv"] Oct 31 13:21:21.626656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.624801 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:21.654659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.647341 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv"] Oct 31 13:21:21.667661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.663790 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:21:21.667661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.663889 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:21:21.667661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.664135 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:21:21.667661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.664209 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:21:21.876800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.862875 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfs9\" (UniqueName: \"kubernetes.io/projected/4f5b81f6-c8b7-4435-8192-8260c0f10a45-kube-api-access-zhfs9\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:21.876800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.862937 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:21.876800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.863188 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:21.995886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.990048 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfs9\" (UniqueName: \"kubernetes.io/projected/4f5b81f6-c8b7-4435-8192-8260c0f10a45-kube-api-access-zhfs9\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:21.995886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.990095 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:21.995886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.990352 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:22.000117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:21.999544 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:22.049944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:22.038568 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfs9\" (UniqueName: \"kubernetes.io/projected/4f5b81f6-c8b7-4435-8192-8260c0f10a45-kube-api-access-zhfs9\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:22.067674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:22.060521 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d63f4f10-2kknv\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:22.300228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:22.300193 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:22.439344 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:21:22.439308 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e45a4d8_ece0_4fbb_9e56_efbb695e362b.slice/crio-4c6f9813a113d669fba4fced674237cdaf1484cde1373dd66e81d0be50e47cb8 WatchSource:0}: Error finding container 4c6f9813a113d669fba4fced674237cdaf1484cde1373dd66e81d0be50e47cb8: Status 404 returned error can't find the container with id 4c6f9813a113d669fba4fced674237cdaf1484cde1373dd66e81d0be50e47cb8 Oct 31 13:21:22.440416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:22.440386 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9"] Oct 31 13:21:22.608076 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:21:22.608028 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5b81f6_c8b7_4435_8192_8260c0f10a45.slice/crio-5b22bf6b85bf89ad0eb81e1547867d95a56f8423a8e172bcae4dc93d50d674f6 WatchSource:0}: Error finding container 5b22bf6b85bf89ad0eb81e1547867d95a56f8423a8e172bcae4dc93d50d674f6: Status 404 returned error can't find the container with id 5b22bf6b85bf89ad0eb81e1547867d95a56f8423a8e172bcae4dc93d50d674f6 Oct 31 13:21:22.610781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:22.610085 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv"] Oct 31 13:21:23.139341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:23.139243 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" event={"ID":"4f5b81f6-c8b7-4435-8192-8260c0f10a45","Type":"ContainerStarted","Data":"7d4aa6173f5c2c441ce29e8a20473f1808fec09fe54fe8d21347061db03c5695"} Oct 31 13:21:23.139341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:23.139304 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" event={"ID":"4f5b81f6-c8b7-4435-8192-8260c0f10a45","Type":"ContainerStarted","Data":"5b22bf6b85bf89ad0eb81e1547867d95a56f8423a8e172bcae4dc93d50d674f6"} Oct 31 13:21:23.141393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:23.141358 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" event={"ID":"5e45a4d8-ece0-4fbb-9e56-efbb695e362b","Type":"ContainerStarted","Data":"65e04ce2ef20b9dfab0bf392a3df81337af65dc0e1cba1b2c1bdb1ec371a5b17"} Oct 31 13:21:23.141525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:23.141421 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" event={"ID":"5e45a4d8-ece0-4fbb-9e56-efbb695e362b","Type":"ContainerStarted","Data":"4c6f9813a113d669fba4fced674237cdaf1484cde1373dd66e81d0be50e47cb8"} Oct 31 13:21:23.141525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:23.141444 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:23.195604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:23.195542 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" podStartSLOduration=2.19552812 podStartE2EDuration="2.19552812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:23.191276243 +0000 UTC m=+2977.368994228" watchObservedRunningTime="2026-10-31 13:21:23.19552812 +0000 UTC m=+2977.373246145" Oct 31 13:21:23.237864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:23.237815 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" podStartSLOduration=2.237801282 podStartE2EDuration="2.237801282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:23.235124123 +0000 UTC m=+2977.412842106" watchObservedRunningTime="2026-10-31 13:21:23.237801282 +0000 UTC m=+2977.415519266" Oct 31 13:21:24.161208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:24.161170 5074 generic.go:356] "Generic (PLEG): container finished" podID="4f5b81f6-c8b7-4435-8192-8260c0f10a45" containerID="7d4aa6173f5c2c441ce29e8a20473f1808fec09fe54fe8d21347061db03c5695" exitCode=0 Oct 31 13:21:24.161657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:24.161228 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" event={"ID":"4f5b81f6-c8b7-4435-8192-8260c0f10a45","Type":"ContainerDied","Data":"7d4aa6173f5c2c441ce29e8a20473f1808fec09fe54fe8d21347061db03c5695"} Oct 31 13:21:25.397029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.396993 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:25.534658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.529857 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-openshift-ca" pod "4f5b81f6-c8b7-4435-8192-8260c0f10a45" (UID: "4f5b81f6-c8b7-4435-8192-8260c0f10a45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:25.557681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.551873 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-openshift-ca\") pod \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " Oct 31 13:21:25.557681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.552270 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-xks-ca\") pod \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " Oct 31 13:21:25.557681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.552374 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5b81f6-c8b7-4435-8192-8260c0f10a45-kube-api-access-zhfs9\" (UniqueName: \"kubernetes.io/projected/4f5b81f6-c8b7-4435-8192-8260c0f10a45-kube-api-access-zhfs9\") pod \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\" (UID: \"4f5b81f6-c8b7-4435-8192-8260c0f10a45\") " Oct 31 13:21:25.631655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.622134 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-xks-ca" pod "4f5b81f6-c8b7-4435-8192-8260c0f10a45" (UID: "4f5b81f6-c8b7-4435-8192-8260c0f10a45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:21:25.631655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.622625 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5b81f6-c8b7-4435-8192-8260c0f10a45-kube-api-access-zhfs9" pod "4f5b81f6-c8b7-4435-8192-8260c0f10a45" (UID: "4f5b81f6-c8b7-4435-8192-8260c0f10a45"). InnerVolumeSpecName "kube-api-access-zhfs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:25.715413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.698294 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:25.715413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.698335 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfs9\" (UniqueName: \"kubernetes.io/projected/4f5b81f6-c8b7-4435-8192-8260c0f10a45-kube-api-access-zhfs9\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:25.715413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:25.698350 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f5b81f6-c8b7-4435-8192-8260c0f10a45-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:25.987381 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:25.987336 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:21:26.183607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:26.183569 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" Oct 31 13:21:26.183829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:26.183606 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv" event={"ID":"4f5b81f6-c8b7-4435-8192-8260c0f10a45","Type":"ContainerDied","Data":"5b22bf6b85bf89ad0eb81e1547867d95a56f8423a8e172bcae4dc93d50d674f6"} Oct 31 13:21:26.183829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:26.183680 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b22bf6b85bf89ad0eb81e1547867d95a56f8423a8e172bcae4dc93d50d674f6" Oct 31 13:21:32.221298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.221255 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn"] Oct 31 13:21:32.222405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.222385 5074 cpu_manager.go:414] "containerMap: removing container" podUID="4f5b81f6-c8b7-4435-8192-8260c0f10a45" containerName="revoke-keys" Oct 31 13:21:32.222462 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.222408 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5b81f6-c8b7-4435-8192-8260c0f10a45" containerName="revoke-keys" Oct 31 13:21:32.222594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.222583 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5b81f6-c8b7-4435-8192-8260c0f10a45" containerName="revoke-keys" Oct 31 13:21:32.227454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.227430 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.234600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.232078 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:21:32.234600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.232144 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:21:32.234600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.232235 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:21:32.234600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.233563 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:21:32.236589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.236564 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn"] Oct 31 13:21:32.242995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.242965 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.243288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.243262 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxq8\" (UniqueName: \"kubernetes.io/projected/2225a273-83c1-44e7-ad99-abb91a537c00-kube-api-access-ftxq8\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.243396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.243355 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.344465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.344427 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.344680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.344613 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxq8\" (UniqueName: \"kubernetes.io/projected/2225a273-83c1-44e7-ad99-abb91a537c00-kube-api-access-ftxq8\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.344781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.344735 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.345395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.345370 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.346817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.346790 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.358330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.358304 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxq8\" (UniqueName: \"kubernetes.io/projected/2225a273-83c1-44e7-ad99-abb91a537c00-kube-api-access-ftxq8\") pod \"maas-api-revoke-sub-df97353d-rxcwn\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.549797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.549138 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:32.894901 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:21:32.894870 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2225a273_83c1_44e7_ad99_abb91a537c00.slice/crio-7077dddaf4b62ad83f3f5cd197b6cef04beedf7c155fb60e7c39038ba1aa657a WatchSource:0}: Error finding container 7077dddaf4b62ad83f3f5cd197b6cef04beedf7c155fb60e7c39038ba1aa657a: Status 404 returned error can't find the container with id 7077dddaf4b62ad83f3f5cd197b6cef04beedf7c155fb60e7c39038ba1aa657a Oct 31 13:21:32.898045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:32.898013 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn"] Oct 31 13:21:33.329126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:33.329086 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" event={"ID":"2225a273-83c1-44e7-ad99-abb91a537c00","Type":"ContainerStarted","Data":"7077dddaf4b62ad83f3f5cd197b6cef04beedf7c155fb60e7c39038ba1aa657a"} Oct 31 13:21:34.061501 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:34.060782 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr"] Oct 31 13:21:34.165279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:34.165197 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:34.347452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:34.347420 5074 generic.go:356] "Generic (PLEG): container finished" podID="2225a273-83c1-44e7-ad99-abb91a537c00" containerID="67e59f368080f1a90159e025e9566980f5d145577e72765779c1a5d90e27bcb2" exitCode=0 Oct 31 13:21:34.347940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:34.347511 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" event={"ID":"2225a273-83c1-44e7-ad99-abb91a537c00","Type":"ContainerDied","Data":"67e59f368080f1a90159e025e9566980f5d145577e72765779c1a5d90e27bcb2"} Oct 31 13:21:35.570442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.570409 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:35.607200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.607105 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh"] Oct 31 13:21:35.609306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.608223 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2225a273-83c1-44e7-ad99-abb91a537c00" containerName="revoke-keys" Oct 31 13:21:35.609306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.608248 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2225a273-83c1-44e7-ad99-abb91a537c00" containerName="revoke-keys" Oct 31 13:21:35.609306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.608487 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2225a273-83c1-44e7-ad99-abb91a537c00" containerName="revoke-keys" Oct 31 13:21:35.613779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.613758 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.619605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.616840 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk"] Oct 31 13:21:35.622059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.620908 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-openshift-ca\") pod \"2225a273-83c1-44e7-ad99-abb91a537c00\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " Oct 31 13:21:35.622059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.620955 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-xks-ca\") pod \"2225a273-83c1-44e7-ad99-abb91a537c00\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " Oct 31 13:21:35.622059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.620993 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2225a273-83c1-44e7-ad99-abb91a537c00-kube-api-access-ftxq8\" (UniqueName: \"kubernetes.io/projected/2225a273-83c1-44e7-ad99-abb91a537c00-kube-api-access-ftxq8\") pod \"2225a273-83c1-44e7-ad99-abb91a537c00\" (UID: \"2225a273-83c1-44e7-ad99-abb91a537c00\") " Oct 31 13:21:35.622280 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.622059 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-openshift-ca" pod "2225a273-83c1-44e7-ad99-abb91a537c00" (UID: "2225a273-83c1-44e7-ad99-abb91a537c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:35.624004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.623523 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.633001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.626757 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh"] Oct 31 13:21:35.633001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.628047 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2225a273-83c1-44e7-ad99-abb91a537c00-kube-api-access-ftxq8" pod "2225a273-83c1-44e7-ad99-abb91a537c00" (UID: "2225a273-83c1-44e7-ad99-abb91a537c00"). InnerVolumeSpecName "kube-api-access-ftxq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:35.633001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.628249 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-xks-ca" pod "2225a273-83c1-44e7-ad99-abb91a537c00" (UID: "2225a273-83c1-44e7-ad99-abb91a537c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:21:35.647780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.647750 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk"] Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726113 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726217 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726268 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726300 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726358 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726400 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726450 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726512 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726554 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726588 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.726666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726615 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ttt\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-kube-api-access-c5ttt\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726691 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726717 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726800 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdhh\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-kube-api-access-ccdhh\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726904 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726928 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.726970 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba5666-3275-407a-90b6-bcd8adb12e64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.727026 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bae7342-6a60-4145-8420-13c5bdf0beb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.727132 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.727148 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225a273-83c1-44e7-ad99-abb91a537c00-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:35.727421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.727163 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxq8\" (UniqueName: \"kubernetes.io/projected/2225a273-83c1-44e7-ad99-abb91a537c00-kube-api-access-ftxq8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:35.828603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828563 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.828814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828607 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.828814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828679 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.828814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828716 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.828814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828745 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.828814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828797 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.828997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828841 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.828997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828872 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.828997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828898 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ttt\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-kube-api-access-c5ttt\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.828997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828949 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.828997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.828972 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.829250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829026 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.829250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829040 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.829250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829053 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdhh\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-kube-api-access-ccdhh\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.829418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829361 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.829549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829522 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.829688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829666 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.829857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829832 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.829927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829843 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.829927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829889 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.829927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829921 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.830040 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.829973 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba5666-3275-407a-90b6-bcd8adb12e64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.830077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.830037 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bae7342-6a60-4145-8420-13c5bdf0beb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.830111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.830098 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.830318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.830276 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.831054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.830978 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bae7342-6a60-4145-8420-13c5bdf0beb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.831054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.830857 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba5666-3275-407a-90b6-bcd8adb12e64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.831054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.830814 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.831425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.831403 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.831748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.831729 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.832458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.832433 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.832898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.832881 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.842386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.842360 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.844608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.844580 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ttt\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-kube-api-access-c5ttt\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:35.845041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.845021 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdhh\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-kube-api-access-ccdhh\") pod \"maas-default-gateway-openshift-default-fc8c56547-nh9mk\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:35.845620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:35.845598 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-fm7lh\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:36.014769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.014731 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:36.032936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.032907 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w"] Oct 31 13:21:36.038531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.038502 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad1a6cf-6mt7w"] Oct 31 13:21:36.054068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.054042 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:36.318677 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:21:36.318614 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ba5666_3275_407a_90b6_bcd8adb12e64.slice/crio-1ba94bf847573afab97ef88367a4169c783b9244b4f635f9d3456df3043021ad WatchSource:0}: Error finding container 1ba94bf847573afab97ef88367a4169c783b9244b4f635f9d3456df3043021ad: Status 404 returned error can't find the container with id 1ba94bf847573afab97ef88367a4169c783b9244b4f635f9d3456df3043021ad Oct 31 13:21:36.318823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.318741 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh"] Oct 31 13:21:36.322046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.322014 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.322129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.322080 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.322129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.322112 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.368258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.368235 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" Oct 31 13:21:36.368394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.368234 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn" event={"ID":"2225a273-83c1-44e7-ad99-abb91a537c00","Type":"ContainerDied","Data":"7077dddaf4b62ad83f3f5cd197b6cef04beedf7c155fb60e7c39038ba1aa657a"} Oct 31 13:21:36.368394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.368366 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7077dddaf4b62ad83f3f5cd197b6cef04beedf7c155fb60e7c39038ba1aa657a" Oct 31 13:21:36.372025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.372001 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" event={"ID":"82ba5666-3275-407a-90b6-bcd8adb12e64","Type":"ContainerStarted","Data":"1ba94bf847573afab97ef88367a4169c783b9244b4f635f9d3456df3043021ad"} Oct 31 13:21:36.388701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.388339 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk"] Oct 31 13:21:36.389877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.389843 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.389988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.389921 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:36.389988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:36.389962 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:21:37.460519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:37.460478 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" event={"ID":"82ba5666-3275-407a-90b6-bcd8adb12e64","Type":"ContainerStarted","Data":"32ba2e750b3936d2830d5b9d43c289df5d2f95dbc39272b55c48564447a01555"} Oct 31 13:21:37.489703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:37.489622 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" event={"ID":"4bae7342-6a60-4145-8420-13c5bdf0beb0","Type":"ContainerStarted","Data":"f4dd03090444c2ad107d1882dd0bd75faae4d22833e096ff10ff68d6ba0a12df"} Oct 31 13:21:37.489910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:37.489727 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" event={"ID":"4bae7342-6a60-4145-8420-13c5bdf0beb0","Type":"ContainerStarted","Data":"771d1018f13c5c9b1ea460843bb2b2217b24a785172ee183eaa907040c123702"} Oct 31 13:21:37.525016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:37.524956 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podStartSLOduration=2.52494293 podStartE2EDuration="2.52494293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:37.516034632 +0000 UTC m=+2991.693752617" watchObservedRunningTime="2026-10-31 13:21:37.52494293 +0000 UTC m=+2991.702660985" Oct 31 13:21:37.550604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:37.550551 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podStartSLOduration=2.550537611 podStartE2EDuration="2.550537611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:37.545406651 +0000 UTC m=+2991.723124636" watchObservedRunningTime="2026-10-31 13:21:37.550537611 +0000 UTC m=+2991.728255595" Oct 31 13:21:37.948001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:37.947970 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03df400-9cc2-41a7-802d-df6728e18264" path="/var/lib/kubelet/pods/b03df400-9cc2-41a7-802d-df6728e18264/volumes" Oct 31 13:21:38.016072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:38.016028 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:38.032654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:38.028784 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:38.032654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:38.028868 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:38.054287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:38.054174 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:38.055608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:38.055579 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:38.055852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:38.055827 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:39.018656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.018451 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:39.018656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.018519 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:39.054700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.054663 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:39.054879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.054724 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:39.062111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.061485 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" podUID="39580112-a39f-4efb-beb7-37f00531978e" containerName="payload-processing" containerID="cri-o://47e6ff853593fc30309d1baee66cf6ef9d95ed46197424db4db4a862e7f10f81" gracePeriod=25 Oct 31 13:21:39.556369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.556286 5074 generic.go:356] "Generic (PLEG): container finished" podID="39580112-a39f-4efb-beb7-37f00531978e" containerID="47e6ff853593fc30309d1baee66cf6ef9d95ed46197424db4db4a862e7f10f81" exitCode=0 Oct 31 13:21:39.557262 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.557223 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" event={"ID":"39580112-a39f-4efb-beb7-37f00531978e","Type":"ContainerDied","Data":"47e6ff853593fc30309d1baee66cf6ef9d95ed46197424db4db4a862e7f10f81"} Oct 31 13:21:39.862061 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.861447 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:21:39.961178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.955874 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-routing-overlay-volume\") pod \"39580112-a39f-4efb-beb7-37f00531978e\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " Oct 31 13:21:39.961178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.956389 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-kube-api-access-7n4rm\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-kube-api-access-7n4rm\") pod \"39580112-a39f-4efb-beb7-37f00531978e\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " Oct 31 13:21:39.961178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.956474 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-provider-credentials\") pod \"39580112-a39f-4efb-beb7-37f00531978e\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " Oct 31 13:21:39.961178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.956717 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-plugins-config-volume\") pod \"39580112-a39f-4efb-beb7-37f00531978e\" (UID: \"39580112-a39f-4efb-beb7-37f00531978e\") " Oct 31 13:21:39.961178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.957887 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-plugins-config-volume" pod "39580112-a39f-4efb-beb7-37f00531978e" (UID: "39580112-a39f-4efb-beb7-37f00531978e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:39.961178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.958333 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-routing-overlay-volume" pod "39580112-a39f-4efb-beb7-37f00531978e" (UID: "39580112-a39f-4efb-beb7-37f00531978e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:39.996657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:39.978057 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-provider-credentials" pod "39580112-a39f-4efb-beb7-37f00531978e" (UID: "39580112-a39f-4efb-beb7-37f00531978e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:40.001389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.001343 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-kube-api-access-7n4rm" pod "39580112-a39f-4efb-beb7-37f00531978e" (UID: "39580112-a39f-4efb-beb7-37f00531978e"). InnerVolumeSpecName "kube-api-access-7n4rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:40.043688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.032789 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:40.043688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.032847 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:40.065006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.055276 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:40.065006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.055350 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:40.065006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.062848 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n4rm\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-kube-api-access-7n4rm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:40.065006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.062883 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39580112-a39f-4efb-beb7-37f00531978e-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:40.065006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.062900 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:40.065006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.062915 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39580112-a39f-4efb-beb7-37f00531978e-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:40.566044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.566015 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" Oct 31 13:21:40.566243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.566044 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr" event={"ID":"39580112-a39f-4efb-beb7-37f00531978e","Type":"ContainerDied","Data":"6a21bd2985c837d6ad3cff5c2880bcdd0df3d3161f618159ad9c4e27008545fd"} Oct 31 13:21:40.566243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.566113 5074 scope.go:122] "RemoveContainer" containerID="47e6ff853593fc30309d1baee66cf6ef9d95ed46197424db4db4a862e7f10f81" Oct 31 13:21:40.641672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.640713 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr"] Oct 31 13:21:40.646372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:40.646338 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-856786ccdc-bmdwr"] Oct 31 13:21:41.024669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:41.022085 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:41.024669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:41.022166 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:41.054479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:41.054443 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:41.054895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:41.054499 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:41.948221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:41.948186 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39580112-a39f-4efb-beb7-37f00531978e" path="/var/lib/kubelet/pods/39580112-a39f-4efb-beb7-37f00531978e/volumes" Oct 31 13:21:42.018657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:42.018096 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:42.018657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:42.018160 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:42.055101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:42.055060 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:42.055499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:42.055123 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:43.023658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:43.022966 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:43.023658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:43.023038 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:43.055057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:43.055022 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:43.055222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:43.055081 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:44.015780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:44.015731 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:44.015995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:44.015822 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:44.054532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:44.054493 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:44.054723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:44.054569 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:44.451332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:44.451227 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9"] Oct 31 13:21:45.051657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:45.050869 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-fm7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 13:21:45.051657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:45.050934 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 13:21:45.100106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:45.086086 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 31 13:21:45.100106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:45.086154 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 31 13:21:46.626249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:46.626217 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:21:46.627102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:46.627078 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:21:46.885144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:46.885052 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:46.886798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:46.886773 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:46.888236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:46.888217 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:21:46.976524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:46.976485 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf"] Oct 31 13:21:46.976769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:46.976703 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" containerID="cri-o://5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06" gracePeriod=30 Oct 31 13:21:47.099788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:47.099742 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-nh9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:21:47.099936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:47.099808 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:21:47.172091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:47.172015 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:47.172564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:47.172542 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:47.174033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:47.174010 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:21:49.453157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:49.453117 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" podUID="5e45a4d8-ece0-4fbb-9e56-efbb695e362b" containerName="payload-processing" containerID="cri-o://65e04ce2ef20b9dfab0bf392a3df81337af65dc0e1cba1b2c1bdb1ec371a5b17" gracePeriod=25 Oct 31 13:21:49.838420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:49.837990 5074 generic.go:356] "Generic (PLEG): container finished" podID="5e45a4d8-ece0-4fbb-9e56-efbb695e362b" containerID="65e04ce2ef20b9dfab0bf392a3df81337af65dc0e1cba1b2c1bdb1ec371a5b17" exitCode=0 Oct 31 13:21:49.838420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:49.838322 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" event={"ID":"5e45a4d8-ece0-4fbb-9e56-efbb695e362b","Type":"ContainerDied","Data":"65e04ce2ef20b9dfab0bf392a3df81337af65dc0e1cba1b2c1bdb1ec371a5b17"} Oct 31 13:21:49.838420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:49.838379 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" event={"ID":"5e45a4d8-ece0-4fbb-9e56-efbb695e362b","Type":"ContainerDied","Data":"4c6f9813a113d669fba4fced674237cdaf1484cde1373dd66e81d0be50e47cb8"} Oct 31 13:21:49.838420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:49.838397 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c6f9813a113d669fba4fced674237cdaf1484cde1373dd66e81d0be50e47cb8" Oct 31 13:21:49.840454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:49.840428 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:50.033972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.033886 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-kube-api-access-dv8j8\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-kube-api-access-dv8j8\") pod \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " Oct 31 13:21:50.034134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.034082 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-plugins-config-volume\") pod \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " Oct 31 13:21:50.034134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.034125 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-provider-credentials\") pod \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " Oct 31 13:21:50.034264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.034244 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-routing-overlay-volume\") pod \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\" (UID: \"5e45a4d8-ece0-4fbb-9e56-efbb695e362b\") " Oct 31 13:21:50.034559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.034530 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-plugins-config-volume" pod "5e45a4d8-ece0-4fbb-9e56-efbb695e362b" (UID: "5e45a4d8-ece0-4fbb-9e56-efbb695e362b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:50.034707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.034621 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-routing-overlay-volume" pod "5e45a4d8-ece0-4fbb-9e56-efbb695e362b" (UID: "5e45a4d8-ece0-4fbb-9e56-efbb695e362b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:50.035080 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.035058 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.035080 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.035080 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.036052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.036030 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-kube-api-access-dv8j8" pod "5e45a4d8-ece0-4fbb-9e56-efbb695e362b" (UID: "5e45a4d8-ece0-4fbb-9e56-efbb695e362b"). InnerVolumeSpecName "kube-api-access-dv8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:50.036531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.036504 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-provider-credentials" pod "5e45a4d8-ece0-4fbb-9e56-efbb695e362b" (UID: "5e45a4d8-ece0-4fbb-9e56-efbb695e362b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:50.137673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.137612 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv8j8\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-kube-api-access-dv8j8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.137673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.137661 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e45a4d8-ece0-4fbb-9e56-efbb695e362b-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:50.846763 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.846730 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9" Oct 31 13:21:50.899954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.899913 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9"] Oct 31 13:21:50.906143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:50.906107 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-54a16b/payload-processing-external-model-e2e-extmod-a-54a16b-84d6gsvp9"] Oct 31 13:21:51.965932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:51.965890 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e45a4d8-ece0-4fbb-9e56-efbb695e362b" path="/var/lib/kubelet/pods/5e45a4d8-ece0-4fbb-9e56-efbb695e362b/volumes" Oct 31 13:21:52.355846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.355779 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 13:21:52.440889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.440845 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-certs\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.440923 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-kube-api-access-hsj2q\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-kube-api-access-hsj2q\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.440995 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230e0627-e62e-4f58-ac8e-91dc89cf2319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230e0627-e62e-4f58-ac8e-91dc89cf2319-istiod-ca-cert\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441139 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-certs" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:52.441252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441205 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-socket\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441242 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-envoy\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441299 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-token\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-token\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441340 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-credential-socket\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441381 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-podinfo\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441418 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-data\") pod \"230e0627-e62e-4f58-ac8e-91dc89cf2319\" (UID: \"230e0627-e62e-4f58-ac8e-91dc89cf2319\") " Oct 31 13:21:52.441479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441454 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230e0627-e62e-4f58-ac8e-91dc89cf2319-istiod-ca-cert" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:21:52.441479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441465 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-socket" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:52.441798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.441768 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-credential-socket" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:52.442055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.442027 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-data" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:52.442279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.442250 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.442279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.442274 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.442425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.442291 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.442425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.442306 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.442425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.442324 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230e0627-e62e-4f58-ac8e-91dc89cf2319-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.443825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.443804 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-podinfo" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:21:52.443999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.443981 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-token" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:52.444114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.444090 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-envoy" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:21:52.445112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.445089 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-kube-api-access-hsj2q" pod "230e0627-e62e-4f58-ac8e-91dc89cf2319" (UID: "230e0627-e62e-4f58-ac8e-91dc89cf2319"). InnerVolumeSpecName "kube-api-access-hsj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:21:52.544465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.544424 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsj2q\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-kube-api-access-hsj2q\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.544465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.544465 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.544735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.544483 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.544735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.544498 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230e0627-e62e-4f58-ac8e-91dc89cf2319-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:21:52.866032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.865986 5074 generic.go:356] "Generic (PLEG): container finished" podID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerID="5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06" exitCode=0 Oct 31 13:21:52.866258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.866064 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" Oct 31 13:21:52.866258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.866072 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" event={"ID":"230e0627-e62e-4f58-ac8e-91dc89cf2319","Type":"ContainerDied","Data":"5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06"} Oct 31 13:21:52.866258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.866137 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf" event={"ID":"230e0627-e62e-4f58-ac8e-91dc89cf2319","Type":"ContainerDied","Data":"c19cda3e8082a6d8b531e0d89cd6264f6c2cd0f7de646da1c7abc506205f3e4c"} Oct 31 13:21:52.866258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.866165 5074 scope.go:122] "RemoveContainer" containerID="5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06" Oct 31 13:21:52.879957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.879939 5074 scope.go:122] "RemoveContainer" containerID="aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f" Oct 31 13:21:52.890605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.890581 5074 scope.go:122] "RemoveContainer" containerID="5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06" Oct 31 13:21:52.890977 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:52.890953 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06\": container with ID starting with 5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06 not found: ID does not exist" containerID="5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06" Oct 31 13:21:52.891078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.890989 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06"} err="failed to get container status \"5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06\": rpc error: code = NotFound desc = could not find container \"5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06\": container with ID starting with 5080abc0a083d26524573afb582b3551036b3f9ac1a06150c23a5d4bf83ded06 not found: ID does not exist" Oct 31 13:21:52.891078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.891014 5074 scope.go:122] "RemoveContainer" containerID="aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f" Oct 31 13:21:52.891316 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:52.891297 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f\": container with ID starting with aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f not found: ID does not exist" containerID="aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f" Oct 31 13:21:52.891357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.891325 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f"} err="failed to get container status \"aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f\": rpc error: code = NotFound desc = could not find container \"aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f\": container with ID starting with aa26db06a5acf218dfcc06ead16df2dec40b6d0991c7a39307a8a6d4eb35354f not found: ID does not exist" Oct 31 13:21:52.902210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.902181 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf"] Oct 31 13:21:52.918388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:52.918357 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-9gxzf"] Oct 31 13:21:53.943312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:21:53.943277 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" path="/var/lib/kubelet/pods/230e0627-e62e-4f58-ac8e-91dc89cf2319/volumes" Oct 31 13:21:55.987153 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:21:55.987095 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:22:03.469392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:03.469350 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:22:07.893797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:07.893748 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:07.980656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:07.979765 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:08.103023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.102977 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:08.137468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.137393 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 13:22:08.140703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.137830 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:22:08.273658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.272736 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:08.273880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.273735 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:08.273954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.273871 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:08.276662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.275411 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:08.413656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.412769 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:08.469813 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:08.469760 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:10.685345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.685300 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47"] Oct 31 13:22:10.686247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686223 5074 cpu_manager.go:414] "containerMap: removing container" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" Oct 31 13:22:10.686247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686248 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" Oct 31 13:22:10.686429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686281 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5e45a4d8-ece0-4fbb-9e56-efbb695e362b" containerName="payload-processing" Oct 31 13:22:10.686429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686290 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e45a4d8-ece0-4fbb-9e56-efbb695e362b" containerName="payload-processing" Oct 31 13:22:10.686429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686319 5074 cpu_manager.go:414] "containerMap: removing container" podUID="39580112-a39f-4efb-beb7-37f00531978e" containerName="payload-processing" Oct 31 13:22:10.686429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686329 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39580112-a39f-4efb-beb7-37f00531978e" containerName="payload-processing" Oct 31 13:22:10.686682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686550 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" Oct 31 13:22:10.686682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686567 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e45a4d8-ece0-4fbb-9e56-efbb695e362b" containerName="payload-processing" Oct 31 13:22:10.686682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.686582 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="39580112-a39f-4efb-beb7-37f00531978e" containerName="payload-processing" Oct 31 13:22:10.692510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.692488 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.706903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.706881 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:22:10.707062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.706888 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:22:10.707062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.706949 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:22:10.707194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.707020 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:22:10.708572 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.708549 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47"] Oct 31 13:22:10.873473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.873439 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.873671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.873540 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.873744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.873681 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88ww\" (UniqueName: \"kubernetes.io/projected/24d7d150-175f-464a-86b6-1384cf8aac4b-kube-api-access-s88ww\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.975034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.974994 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.975215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.975076 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.975215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.975200 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88ww\" (UniqueName: \"kubernetes.io/projected/24d7d150-175f-464a-86b6-1384cf8aac4b-kube-api-access-s88ww\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.976105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.976077 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.977943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.977917 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:10.996067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:10.996030 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88ww\" (UniqueName: \"kubernetes.io/projected/24d7d150-175f-464a-86b6-1384cf8aac4b-kube-api-access-s88ww\") pod \"maas-api-revoke-sub-0c3da27e-skz47\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:11.013750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:11.013726 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:11.311235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:11.311200 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47"] Oct 31 13:22:11.311434 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:22:11.311407 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d7d150_175f_464a_86b6_1384cf8aac4b.slice/crio-206474b300f03d65eb4c53351849237bec36000a508be58009c66024dd04e987 WatchSource:0}: Error finding container 206474b300f03d65eb4c53351849237bec36000a508be58009c66024dd04e987: Status 404 returned error can't find the container with id 206474b300f03d65eb4c53351849237bec36000a508be58009c66024dd04e987 Oct 31 13:22:12.139911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:12.139873 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" event={"ID":"24d7d150-175f-464a-86b6-1384cf8aac4b","Type":"ContainerStarted","Data":"0452e5d97756284d6bd488c8cb156faa8631358f129338699e68dee39c236337"} Oct 31 13:22:12.140330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:12.139936 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" event={"ID":"24d7d150-175f-464a-86b6-1384cf8aac4b","Type":"ContainerStarted","Data":"206474b300f03d65eb4c53351849237bec36000a508be58009c66024dd04e987"} Oct 31 13:22:13.149429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:13.149388 5074 generic.go:356] "Generic (PLEG): container finished" podID="24d7d150-175f-464a-86b6-1384cf8aac4b" containerID="0452e5d97756284d6bd488c8cb156faa8631358f129338699e68dee39c236337" exitCode=0 Oct 31 13:22:13.149917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:13.149466 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" event={"ID":"24d7d150-175f-464a-86b6-1384cf8aac4b","Type":"ContainerDied","Data":"0452e5d97756284d6bd488c8cb156faa8631358f129338699e68dee39c236337"} Oct 31 13:22:14.349648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.349607 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:14.564287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.564248 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-openshift-ca\") pod \"24d7d150-175f-464a-86b6-1384cf8aac4b\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " Oct 31 13:22:14.564452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.564342 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-xks-ca\") pod \"24d7d150-175f-464a-86b6-1384cf8aac4b\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " Oct 31 13:22:14.564452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.564437 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d7d150-175f-464a-86b6-1384cf8aac4b-kube-api-access-s88ww\" (UniqueName: \"kubernetes.io/projected/24d7d150-175f-464a-86b6-1384cf8aac4b-kube-api-access-s88ww\") pod \"24d7d150-175f-464a-86b6-1384cf8aac4b\" (UID: \"24d7d150-175f-464a-86b6-1384cf8aac4b\") " Oct 31 13:22:14.564706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.564667 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-openshift-ca" pod "24d7d150-175f-464a-86b6-1384cf8aac4b" (UID: "24d7d150-175f-464a-86b6-1384cf8aac4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:14.565105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.565081 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:14.566567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.566532 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-xks-ca" pod "24d7d150-175f-464a-86b6-1384cf8aac4b" (UID: "24d7d150-175f-464a-86b6-1384cf8aac4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:22:14.566709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.566683 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d7d150-175f-464a-86b6-1384cf8aac4b-kube-api-access-s88ww" pod "24d7d150-175f-464a-86b6-1384cf8aac4b" (UID: "24d7d150-175f-464a-86b6-1384cf8aac4b"). InnerVolumeSpecName "kube-api-access-s88ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:14.667212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.667164 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7d150-175f-464a-86b6-1384cf8aac4b-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:14.667212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:14.667209 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88ww\" (UniqueName: \"kubernetes.io/projected/24d7d150-175f-464a-86b6-1384cf8aac4b-kube-api-access-s88ww\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:15.167414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:15.167384 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" Oct 31 13:22:15.167609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:15.167421 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47" event={"ID":"24d7d150-175f-464a-86b6-1384cf8aac4b","Type":"ContainerDied","Data":"206474b300f03d65eb4c53351849237bec36000a508be58009c66024dd04e987"} Oct 31 13:22:15.167609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:15.167474 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="206474b300f03d65eb4c53351849237bec36000a508be58009c66024dd04e987" Oct 31 13:22:21.175836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.175792 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l"] Oct 31 13:22:21.176596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.176580 5074 cpu_manager.go:414] "containerMap: removing container" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" Oct 31 13:22:21.176667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.176597 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" Oct 31 13:22:21.176667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.176645 5074 cpu_manager.go:414] "containerMap: removing container" podUID="24d7d150-175f-464a-86b6-1384cf8aac4b" containerName="revoke-keys" Oct 31 13:22:21.176667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.176651 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d7d150-175f-464a-86b6-1384cf8aac4b" containerName="revoke-keys" Oct 31 13:22:21.176850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.176839 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="230e0627-e62e-4f58-ac8e-91dc89cf2319" containerName="istio-proxy" Oct 31 13:22:21.176894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.176854 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d7d150-175f-464a-86b6-1384cf8aac4b" containerName="revoke-keys" Oct 31 13:22:21.181070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.181043 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.192070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.192032 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l"] Oct 31 13:22:21.196433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.196395 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:22:21.196803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.196780 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:22:21.196968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.196897 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:22:21.196968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.196927 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:22:21.201286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.201257 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.201687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.201663 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.201902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.201884 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qvn\" (UniqueName: \"kubernetes.io/projected/144dce9f-4826-4397-8e96-84ce54f24c3c-kube-api-access-k2qvn\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.303356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.303316 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.303550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.303451 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.303550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.303503 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qvn\" (UniqueName: \"kubernetes.io/projected/144dce9f-4826-4397-8e96-84ce54f24c3c-kube-api-access-k2qvn\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.304202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.304175 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.305982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.305948 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.318411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.318382 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qvn\" (UniqueName: \"kubernetes.io/projected/144dce9f-4826-4397-8e96-84ce54f24c3c-kube-api-access-k2qvn\") pod \"maas-api-revoke-sub-9c057fdf-dmg7l\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.513744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.513706 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:21.805811 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:22:21.805777 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144dce9f_4826_4397_8e96_84ce54f24c3c.slice/crio-82ad26897c5c41cb66574e70c07e7e25a3d048c2cdb052d4b38d4383293ead03 WatchSource:0}: Error finding container 82ad26897c5c41cb66574e70c07e7e25a3d048c2cdb052d4b38d4383293ead03: Status 404 returned error can't find the container with id 82ad26897c5c41cb66574e70c07e7e25a3d048c2cdb052d4b38d4383293ead03 Oct 31 13:22:21.807574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:21.807535 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l"] Oct 31 13:22:22.238962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:22.238920 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" event={"ID":"144dce9f-4826-4397-8e96-84ce54f24c3c","Type":"ContainerStarted","Data":"e960757077826e6ea651776b9cd91534add634df39c963752fff3d5419976c5a"} Oct 31 13:22:22.239371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:22.238978 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" event={"ID":"144dce9f-4826-4397-8e96-84ce54f24c3c","Type":"ContainerStarted","Data":"82ad26897c5c41cb66574e70c07e7e25a3d048c2cdb052d4b38d4383293ead03"} Oct 31 13:22:22.262644 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:22.262585 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" podStartSLOduration=1.262572748 podStartE2EDuration="1.262572748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:22.258662369 +0000 UTC m=+3036.436380352" watchObservedRunningTime="2026-10-31 13:22:22.262572748 +0000 UTC m=+3036.440290767" Oct 31 13:22:23.249989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:23.249958 5074 generic.go:356] "Generic (PLEG): container finished" podID="144dce9f-4826-4397-8e96-84ce54f24c3c" containerID="e960757077826e6ea651776b9cd91534add634df39c963752fff3d5419976c5a" exitCode=0 Oct 31 13:22:23.250392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:23.250047 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" event={"ID":"144dce9f-4826-4397-8e96-84ce54f24c3c","Type":"ContainerDied","Data":"e960757077826e6ea651776b9cd91534add634df39c963752fff3d5419976c5a"} Oct 31 13:22:24.487888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.487853 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:24.600705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.599944 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144dce9f-4826-4397-8e96-84ce54f24c3c-kube-api-access-k2qvn\" (UniqueName: \"kubernetes.io/projected/144dce9f-4826-4397-8e96-84ce54f24c3c-kube-api-access-k2qvn\") pod \"144dce9f-4826-4397-8e96-84ce54f24c3c\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " Oct 31 13:22:24.600705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.600364 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-xks-ca\") pod \"144dce9f-4826-4397-8e96-84ce54f24c3c\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " Oct 31 13:22:24.600705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.600433 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-openshift-ca\") pod \"144dce9f-4826-4397-8e96-84ce54f24c3c\" (UID: \"144dce9f-4826-4397-8e96-84ce54f24c3c\") " Oct 31 13:22:24.648658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.647883 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-openshift-ca" pod "144dce9f-4826-4397-8e96-84ce54f24c3c" (UID: "144dce9f-4826-4397-8e96-84ce54f24c3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:24.677662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.673786 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-xks-ca" pod "144dce9f-4826-4397-8e96-84ce54f24c3c" (UID: "144dce9f-4826-4397-8e96-84ce54f24c3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:22:24.822726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.794420 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:24.822726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.794457 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144dce9f-4826-4397-8e96-84ce54f24c3c-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:24.822726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.810973 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144dce9f-4826-4397-8e96-84ce54f24c3c-kube-api-access-k2qvn" pod "144dce9f-4826-4397-8e96-84ce54f24c3c" (UID: "144dce9f-4826-4397-8e96-84ce54f24c3c"). InnerVolumeSpecName "kube-api-access-k2qvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:24.911699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:24.910873 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2qvn\" (UniqueName: \"kubernetes.io/projected/144dce9f-4826-4397-8e96-84ce54f24c3c-kube-api-access-k2qvn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:25.295658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:25.295341 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" event={"ID":"144dce9f-4826-4397-8e96-84ce54f24c3c","Type":"ContainerDied","Data":"82ad26897c5c41cb66574e70c07e7e25a3d048c2cdb052d4b38d4383293ead03"} Oct 31 13:22:25.295658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:25.295413 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82ad26897c5c41cb66574e70c07e7e25a3d048c2cdb052d4b38d4383293ead03" Oct 31 13:22:25.295658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:25.295488 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l" Oct 31 13:22:26.112414 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:22:26.107087 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:22:27.269658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:27.268755 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:27.276736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:27.276698 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 13:22:27.411879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:27.411829 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:27.413258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:27.413229 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:22:27.413471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:27.413446 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:27.869771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:27.869715 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:22:27.941007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:27.939915 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 13:22:33.350132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:33.349931 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt"] Oct 31 13:22:38.351449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.351400 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" podUID="a57421ac-c611-4bee-a0fe-a7e4b143b00a" containerName="payload-processing" containerID="cri-o://7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5" gracePeriod=25 Oct 31 13:22:38.718598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.718561 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:22:38.720176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.720150 5074 generic.go:356] "Generic (PLEG): container finished" podID="a57421ac-c611-4bee-a0fe-a7e4b143b00a" containerID="7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5" exitCode=0 Oct 31 13:22:38.720331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.720199 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" Oct 31 13:22:38.720331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.720230 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" event={"ID":"a57421ac-c611-4bee-a0fe-a7e4b143b00a","Type":"ContainerDied","Data":"7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5"} Oct 31 13:22:38.720331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.720295 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt" event={"ID":"a57421ac-c611-4bee-a0fe-a7e4b143b00a","Type":"ContainerDied","Data":"61df2449b4b9fb89648cbd8adcf8906580cf55402a9409674437d3d59a4a8b91"} Oct 31 13:22:38.720331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.720320 5074 scope.go:122] "RemoveContainer" containerID="7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5" Oct 31 13:22:38.731455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.731437 5074 scope.go:122] "RemoveContainer" containerID="7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5" Oct 31 13:22:38.731747 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:22:38.731726 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5\": container with ID starting with 7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5 not found: ID does not exist" containerID="7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5" Oct 31 13:22:38.731830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.731759 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5"} err="failed to get container status \"7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5\": rpc error: code = NotFound desc = could not find container \"7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5\": container with ID starting with 7bbedeff6b63d460c8e3c3c00e74ac4debdf420a287f47756f45237d53646bb5 not found: ID does not exist" Oct 31 13:22:38.819242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.819191 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-kube-api-access-q68pm\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-kube-api-access-q68pm\") pod \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " Oct 31 13:22:38.819462 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.819444 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-routing-overlay-volume\") pod \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " Oct 31 13:22:38.819561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.819531 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-provider-credentials\") pod \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " Oct 31 13:22:38.819741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.819724 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-plugins-config-volume\") pod \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\" (UID: \"a57421ac-c611-4bee-a0fe-a7e4b143b00a\") " Oct 31 13:22:38.819824 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.819782 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-routing-overlay-volume" pod "a57421ac-c611-4bee-a0fe-a7e4b143b00a" (UID: "a57421ac-c611-4bee-a0fe-a7e4b143b00a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:38.820187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.820157 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-plugins-config-volume" pod "a57421ac-c611-4bee-a0fe-a7e4b143b00a" (UID: "a57421ac-c611-4bee-a0fe-a7e4b143b00a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:22:38.820448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.820426 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:38.820492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.820456 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a57421ac-c611-4bee-a0fe-a7e4b143b00a-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:38.821588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.821557 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-kube-api-access-q68pm" pod "a57421ac-c611-4bee-a0fe-a7e4b143b00a" (UID: "a57421ac-c611-4bee-a0fe-a7e4b143b00a"). InnerVolumeSpecName "kube-api-access-q68pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:38.821843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.821821 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-provider-credentials" pod "a57421ac-c611-4bee-a0fe-a7e4b143b00a" (UID: "a57421ac-c611-4bee-a0fe-a7e4b143b00a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:22:38.922484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.922453 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q68pm\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-kube-api-access-q68pm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:38.922484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:38.922484 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a57421ac-c611-4bee-a0fe-a7e4b143b00a-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:22:39.036343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:39.036317 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr"] Oct 31 13:22:39.041323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:39.041286 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23b63abe-jv2xr"] Oct 31 13:22:39.050512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:39.050485 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt"] Oct 31 13:22:39.058012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:39.057983 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-59c6fjdqt"] Oct 31 13:22:39.940744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:39.940708 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57421ac-c611-4bee-a0fe-a7e4b143b00a" path="/var/lib/kubelet/pods/a57421ac-c611-4bee-a0fe-a7e4b143b00a/volumes" Oct 31 13:22:39.941303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:39.941286 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8e1ebd-c9e0-497b-b941-b7d64edc141e" path="/var/lib/kubelet/pods/ff8e1ebd-c9e0-497b-b941-b7d64edc141e/volumes" Oct 31 13:22:45.167719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:45.167683 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:22:51.040658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.039683 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm"] Oct 31 13:22:51.106071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.106020 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e0cbc36-fz4rm"] Oct 31 13:22:51.595244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.595206 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r"] Oct 31 13:22:51.596064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.596038 5074 cpu_manager.go:414] "containerMap: removing container" podUID="144dce9f-4826-4397-8e96-84ce54f24c3c" containerName="revoke-keys" Oct 31 13:22:51.596064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.596063 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144dce9f-4826-4397-8e96-84ce54f24c3c" containerName="revoke-keys" Oct 31 13:22:51.596249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.596099 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a57421ac-c611-4bee-a0fe-a7e4b143b00a" containerName="payload-processing" Oct 31 13:22:51.596249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.596107 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57421ac-c611-4bee-a0fe-a7e4b143b00a" containerName="payload-processing" Oct 31 13:22:51.596367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.596330 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="144dce9f-4826-4397-8e96-84ce54f24c3c" containerName="revoke-keys" Oct 31 13:22:51.596367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.596350 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57421ac-c611-4bee-a0fe-a7e4b143b00a" containerName="payload-processing" Oct 31 13:22:51.600533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.600514 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.620753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.611585 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"agc-cred-d0bf873b9006\"" Oct 31 13:22:51.620753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.611722 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"routing-overlay\"" Oct 31 13:22:51.620753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.611780 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ef744\"" Oct 31 13:22:51.620753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.614850 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"kube-root-ca.crt\"" Oct 31 13:22:51.629045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.626750 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r"] Oct 31 13:22:51.707267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.707225 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ef744\"/\"openshift-service-ca.crt\"" Oct 31 13:22:51.708036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.708008 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.708170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.708154 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.708313 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.708300 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.708362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.708351 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7lx\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-kube-api-access-lq7lx\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.809943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.809900 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7lx\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-kube-api-access-lq7lx\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.810142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.810052 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.810213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.810173 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.810389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.810370 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.810894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.810867 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.811233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.811208 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.812848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.812822 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:51.896270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.896075 5074 scope.go:122] "RemoveContainer" containerID="c5062b315f2c3b228828fc05e90a1897c9211d29d96499474de6d59155909854" Oct 31 13:22:51.913032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:51.901408 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7lx\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-kube-api-access-lq7lx\") pod \"payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:52.038724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:52.038683 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7276306b-3f0b-49d8-8c54-878c666ac783" path="/var/lib/kubelet/pods/7276306b-3f0b-49d8-8c54-878c666ac783/volumes" Oct 31 13:22:52.482659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:52.160912 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:53.028351 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:22:53.028314 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b14eaab_3a3e_4290_befd_88264e37198b.slice/crio-4616e4b58b5b6a8ff0139210b7514d9d8df2226772742df9918b31c88a768045 WatchSource:0}: Error finding container 4616e4b58b5b6a8ff0139210b7514d9d8df2226772742df9918b31c88a768045: Status 404 returned error can't find the container with id 4616e4b58b5b6a8ff0139210b7514d9d8df2226772742df9918b31c88a768045 Oct 31 13:22:53.030391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:53.030358 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r"] Oct 31 13:22:53.543994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:53.543952 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" event={"ID":"1b14eaab-3a3e-4290-befd-88264e37198b","Type":"ContainerStarted","Data":"a191a86d45d6daedfed45feba2e862c24c1a17470fdd1e104744d69041500d6e"} Oct 31 13:22:53.544443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:53.544020 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" event={"ID":"1b14eaab-3a3e-4290-befd-88264e37198b","Type":"ContainerStarted","Data":"4616e4b58b5b6a8ff0139210b7514d9d8df2226772742df9918b31c88a768045"} Oct 31 13:22:53.544443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:53.544055 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:22:53.566306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:53.566252 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" podStartSLOduration=2.566238353 podStartE2EDuration="2.566238353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:53.56326957 +0000 UTC m=+3067.740987555" watchObservedRunningTime="2026-10-31 13:22:53.566238353 +0000 UTC m=+3067.743956337" Oct 31 13:22:55.988165 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:22:55.988119 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:22:59.083405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:59.083371 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l"] Oct 31 13:22:59.090957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:59.090923 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb68f83-clk4l"] Oct 31 13:22:59.941344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:22:59.941308 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d78e531-db74-427e-8bc9-2760cc87c9a8" path="/var/lib/kubelet/pods/9d78e531-db74-427e-8bc9-2760cc87c9a8/volumes" Oct 31 13:23:04.555556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:04.555510 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:23:10.041511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:10.041474 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn"] Oct 31 13:23:10.049123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:10.049095 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc20555-bqxsn"] Oct 31 13:23:11.940373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:11.940336 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4803445e-6131-4e74-a493-d74dda90b945" path="/var/lib/kubelet/pods/4803445e-6131-4e74-a493-d74dda90b945/volumes" Oct 31 13:23:17.551502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:17.551459 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r"] Oct 31 13:23:22.053626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:22.053573 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5"] Oct 31 13:23:22.062962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:22.062927 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8fd992d-6brd5"] Oct 31 13:23:22.556657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:22.555724 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" podUID="1b14eaab-3a3e-4290-befd-88264e37198b" containerName="payload-processing" containerID="cri-o://a191a86d45d6daedfed45feba2e862c24c1a17470fdd1e104744d69041500d6e" gracePeriod=25 Oct 31 13:23:22.992437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:22.992282 5074 generic.go:356] "Generic (PLEG): container finished" podID="1b14eaab-3a3e-4290-befd-88264e37198b" containerID="a191a86d45d6daedfed45feba2e862c24c1a17470fdd1e104744d69041500d6e" exitCode=0 Oct 31 13:23:22.992437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:22.992357 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" event={"ID":"1b14eaab-3a3e-4290-befd-88264e37198b","Type":"ContainerDied","Data":"a191a86d45d6daedfed45feba2e862c24c1a17470fdd1e104744d69041500d6e"} Oct 31 13:23:23.109917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.109884 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:23:23.252228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.252188 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-provider-credentials\") pod \"1b14eaab-3a3e-4290-befd-88264e37198b\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " Oct 31 13:23:23.253510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.253483 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-routing-overlay-volume\") pod \"1b14eaab-3a3e-4290-befd-88264e37198b\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " Oct 31 13:23:23.254303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.254281 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-plugins-config-volume\") pod \"1b14eaab-3a3e-4290-befd-88264e37198b\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " Oct 31 13:23:23.254957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.254937 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-kube-api-access-lq7lx\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-kube-api-access-lq7lx\") pod \"1b14eaab-3a3e-4290-befd-88264e37198b\" (UID: \"1b14eaab-3a3e-4290-befd-88264e37198b\") " Oct 31 13:23:23.256179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.254083 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-routing-overlay-volume" pod "1b14eaab-3a3e-4290-befd-88264e37198b" (UID: "1b14eaab-3a3e-4290-befd-88264e37198b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:23.256325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.254858 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-plugins-config-volume" pod "1b14eaab-3a3e-4290-befd-88264e37198b" (UID: "1b14eaab-3a3e-4290-befd-88264e37198b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:23.265142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.265103 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-kube-api-access-lq7lx" pod "1b14eaab-3a3e-4290-befd-88264e37198b" (UID: "1b14eaab-3a3e-4290-befd-88264e37198b"). InnerVolumeSpecName "kube-api-access-lq7lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:23.292927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.292873 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-provider-credentials" pod "1b14eaab-3a3e-4290-befd-88264e37198b" (UID: "1b14eaab-3a3e-4290-befd-88264e37198b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:23.363948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.362202 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:23.363948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.362255 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq7lx\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-kube-api-access-lq7lx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:23.363948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.362274 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b14eaab-3a3e-4290-befd-88264e37198b-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:23.363948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:23.362290 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b14eaab-3a3e-4290-befd-88264e37198b-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:24.016503 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:24.016461 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e94e2e-bdbb-423f-99e8-5fa920cfad10" path="/var/lib/kubelet/pods/94e94e2e-bdbb-423f-99e8-5fa920cfad10/volumes" Oct 31 13:23:24.052671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:24.026737 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" event={"ID":"1b14eaab-3a3e-4290-befd-88264e37198b","Type":"ContainerDied","Data":"4616e4b58b5b6a8ff0139210b7514d9d8df2226772742df9918b31c88a768045"} Oct 31 13:23:24.053018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:24.052987 5074 scope.go:122] "RemoveContainer" containerID="a191a86d45d6daedfed45feba2e862c24c1a17470fdd1e104744d69041500d6e" Oct 31 13:23:24.053352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:24.016589 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r" Oct 31 13:23:24.169946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:24.169704 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r"] Oct 31 13:23:24.175825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:24.175742 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ef744/payload-processing-external-model-e2e-extmod-a-9ef744-56dfbzp6r"] Oct 31 13:23:25.924662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:25.923775 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:25.973658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:25.969827 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:26.048728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:26.043526 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b14eaab-3a3e-4290-befd-88264e37198b" path="/var/lib/kubelet/pods/1b14eaab-3a3e-4290-befd-88264e37198b/volumes" Oct 31 13:23:26.140657 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:23:26.140598 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:23:27.030092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:27.030049 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9"] Oct 31 13:23:27.034695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:27.034663 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2267da-xmhn9"] Oct 31 13:23:27.973197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:27.973156 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844d1780-a71f-4847-92c3-7fac1ce625c3" path="/var/lib/kubelet/pods/844d1780-a71f-4847-92c3-7fac1ce625c3/volumes" Oct 31 13:23:28.229260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.229182 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn"] Oct 31 13:23:28.229973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.229955 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1b14eaab-3a3e-4290-befd-88264e37198b" containerName="payload-processing" Oct 31 13:23:28.230057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.229974 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b14eaab-3a3e-4290-befd-88264e37198b" containerName="payload-processing" Oct 31 13:23:28.231883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.230916 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b14eaab-3a3e-4290-befd-88264e37198b" containerName="payload-processing" Oct 31 13:23:28.238148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.238120 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.243184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.243154 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:23:28.244016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.243750 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:23:28.244016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.243832 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:23:28.244858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.244831 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:23:28.253323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.253297 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn"] Oct 31 13:23:28.253911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.253889 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.254035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.253982 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.254310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.254292 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwl8p\" (UniqueName: \"kubernetes.io/projected/28f27cd2-47ab-4259-91c3-83a951f9d96f-kube-api-access-dwl8p\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.319780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.319739 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db"] Oct 31 13:23:28.324768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.324741 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.342444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.342417 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db"] Oct 31 13:23:28.355704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.355675 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.355865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.355715 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2f67\" (UniqueName: \"kubernetes.io/projected/24956f45-acac-405b-b473-5850ab98e464-kube-api-access-m2f67\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.355942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.355889 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwl8p\" (UniqueName: \"kubernetes.io/projected/28f27cd2-47ab-4259-91c3-83a951f9d96f-kube-api-access-dwl8p\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.355942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.355916 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24956f45-acac-405b-b473-5850ab98e464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.356073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.356016 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24956f45-acac-405b-b473-5850ab98e464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.356073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.356054 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.356818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.356794 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.358864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.358836 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.372587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.372562 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwl8p\" (UniqueName: \"kubernetes.io/projected/28f27cd2-47ab-4259-91c3-83a951f9d96f-kube-api-access-dwl8p\") pod \"maas-api-revoke-sub-49221240-jsdcn\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.457096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.457058 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24956f45-acac-405b-b473-5850ab98e464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.457278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.457161 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24956f45-acac-405b-b473-5850ab98e464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.457278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.457231 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2f67\" (UniqueName: \"kubernetes.io/projected/24956f45-acac-405b-b473-5850ab98e464-kube-api-access-m2f67\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.457870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.457849 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24956f45-acac-405b-b473-5850ab98e464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.459495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.459472 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24956f45-acac-405b-b473-5850ab98e464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.470239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.470208 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2f67\" (UniqueName: \"kubernetes.io/projected/24956f45-acac-405b-b473-5850ab98e464-kube-api-access-m2f67\") pod \"maas-api-revoke-sub-9be9cb68-vt5db\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.560049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.559960 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:28.650982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.649847 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:28.846976 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:23:28.846945 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f27cd2_47ab_4259_91c3_83a951f9d96f.slice/crio-c53e1447f808db3527e6d50d6ed2b9fea384eb2003c6afa5692f5368c42164b6 WatchSource:0}: Error finding container c53e1447f808db3527e6d50d6ed2b9fea384eb2003c6afa5692f5368c42164b6: Status 404 returned error can't find the container with id c53e1447f808db3527e6d50d6ed2b9fea384eb2003c6afa5692f5368c42164b6 Oct 31 13:23:28.847297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.847271 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn"] Oct 31 13:23:28.993112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:28.993082 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db"] Oct 31 13:23:29.078753 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:23:29.074202 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24956f45_acac_405b_b473_5850ab98e464.slice/crio-c12d1a342674c77b01d672d078da1baa476d4a9064b6808a4dab85d6a4564eb4 WatchSource:0}: Error finding container c12d1a342674c77b01d672d078da1baa476d4a9064b6808a4dab85d6a4564eb4: Status 404 returned error can't find the container with id c12d1a342674c77b01d672d078da1baa476d4a9064b6808a4dab85d6a4564eb4 Oct 31 13:23:29.317153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.316944 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" event={"ID":"24956f45-acac-405b-b473-5850ab98e464","Type":"ContainerStarted","Data":"c12d1a342674c77b01d672d078da1baa476d4a9064b6808a4dab85d6a4564eb4"} Oct 31 13:23:29.333747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.333647 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" event={"ID":"28f27cd2-47ab-4259-91c3-83a951f9d96f","Type":"ContainerStarted","Data":"c53e1447f808db3527e6d50d6ed2b9fea384eb2003c6afa5692f5368c42164b6"} Oct 31 13:23:29.813704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.813668 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99"] Oct 31 13:23:29.818594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.818569 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:29.827581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.827550 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99"] Oct 31 13:23:29.827795 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.827603 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"openshift-service-ca.crt\"" Oct 31 13:23:29.827795 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.827603 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5a9e45\"" Oct 31 13:23:29.827795 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.827628 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"kube-root-ca.crt\"" Oct 31 13:23:29.828078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.828058 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"agc-cred-858924e837d5\"" Oct 31 13:23:29.828078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.828062 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"routing-overlay\"" Oct 31 13:23:29.828558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:29.828542 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"payload-processing-external-model-e2e--527e58cd-dockercfg-dvz52\"" Oct 31 13:23:30.072986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.052465 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.072986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.052691 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dprb\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-kube-api-access-7dprb\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.072986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.052728 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.072986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.052769 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.200565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.199886 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.200565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.200038 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dprb\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-kube-api-access-7dprb\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.214074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.200073 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.214074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.200938 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.214074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.202523 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.214074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.203554 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.340428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.338074 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.385150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.385115 5074 generic.go:356] "Generic (PLEG): container finished" podID="24956f45-acac-405b-b473-5850ab98e464" containerID="916f1c8c28cf8354cd96b4b2b43b571d4a5df693f2e0048d292a38d3b5b95938" exitCode=0 Oct 31 13:23:30.385301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.385239 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" event={"ID":"24956f45-acac-405b-b473-5850ab98e464","Type":"ContainerDied","Data":"916f1c8c28cf8354cd96b4b2b43b571d4a5df693f2e0048d292a38d3b5b95938"} Oct 31 13:23:30.385428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.385400 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dprb\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-kube-api-access-7dprb\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.387188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.387163 5074 generic.go:356] "Generic (PLEG): container finished" podID="28f27cd2-47ab-4259-91c3-83a951f9d96f" containerID="6b15b5e437f2d0a288f99e6d87503ddbd6f0f11619c7d7f787fbfa887a4c80ad" exitCode=0 Oct 31 13:23:30.387306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.387198 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" event={"ID":"28f27cd2-47ab-4259-91c3-83a951f9d96f","Type":"ContainerDied","Data":"6b15b5e437f2d0a288f99e6d87503ddbd6f0f11619c7d7f787fbfa887a4c80ad"} Oct 31 13:23:30.438905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.438865 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:30.730839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:30.730805 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99"] Oct 31 13:23:31.400442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.400398 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" event={"ID":"199d1ca7-a6fc-43ce-9392-216fdef9f347","Type":"ContainerStarted","Data":"7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2"} Oct 31 13:23:31.401003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.400502 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" event={"ID":"199d1ca7-a6fc-43ce-9392-216fdef9f347","Type":"ContainerStarted","Data":"042bd8ee8e9050c932bec3f8430774eb998015c6b95ef8d6a1b5d5ff596c1c49"} Oct 31 13:23:31.401003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.400539 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:31.422128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.422074 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" podStartSLOduration=2.422059115 podStartE2EDuration="2.422059115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:31.420820254 +0000 UTC m=+3105.598538238" watchObservedRunningTime="2026-10-31 13:23:31.422059115 +0000 UTC m=+3105.599777109" Oct 31 13:23:31.810096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.810064 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:31.817247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.817221 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:31.907965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.907930 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f27cd2-47ab-4259-91c3-83a951f9d96f-kube-api-access-dwl8p\" (UniqueName: \"kubernetes.io/projected/28f27cd2-47ab-4259-91c3-83a951f9d96f-kube-api-access-dwl8p\") pod \"28f27cd2-47ab-4259-91c3-83a951f9d96f\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " Oct 31 13:23:31.908176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.908024 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-openshift-ca\") pod \"28f27cd2-47ab-4259-91c3-83a951f9d96f\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " Oct 31 13:23:31.908176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.908053 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24956f45-acac-405b-b473-5850ab98e464-kube-api-access-m2f67\" (UniqueName: \"kubernetes.io/projected/24956f45-acac-405b-b473-5850ab98e464-kube-api-access-m2f67\") pod \"24956f45-acac-405b-b473-5850ab98e464\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " Oct 31 13:23:31.908176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.908085 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24956f45-acac-405b-b473-5850ab98e464-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24956f45-acac-405b-b473-5850ab98e464-maas-api-xks-ca\") pod \"24956f45-acac-405b-b473-5850ab98e464\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " Oct 31 13:23:31.908349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.908188 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-xks-ca\") pod \"28f27cd2-47ab-4259-91c3-83a951f9d96f\" (UID: \"28f27cd2-47ab-4259-91c3-83a951f9d96f\") " Oct 31 13:23:31.908428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.908410 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24956f45-acac-405b-b473-5850ab98e464-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24956f45-acac-405b-b473-5850ab98e464-maas-api-openshift-ca\") pod \"24956f45-acac-405b-b473-5850ab98e464\" (UID: \"24956f45-acac-405b-b473-5850ab98e464\") " Oct 31 13:23:31.910159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.910127 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-openshift-ca" pod "28f27cd2-47ab-4259-91c3-83a951f9d96f" (UID: "28f27cd2-47ab-4259-91c3-83a951f9d96f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:31.910318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.910297 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24956f45-acac-405b-b473-5850ab98e464-maas-api-openshift-ca" pod "24956f45-acac-405b-b473-5850ab98e464" (UID: "24956f45-acac-405b-b473-5850ab98e464"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:31.910518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.910489 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f27cd2-47ab-4259-91c3-83a951f9d96f-kube-api-access-dwl8p" pod "28f27cd2-47ab-4259-91c3-83a951f9d96f" (UID: "28f27cd2-47ab-4259-91c3-83a951f9d96f"). InnerVolumeSpecName "kube-api-access-dwl8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:31.911040 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.911013 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24956f45-acac-405b-b473-5850ab98e464-kube-api-access-m2f67" pod "24956f45-acac-405b-b473-5850ab98e464" (UID: "24956f45-acac-405b-b473-5850ab98e464"). InnerVolumeSpecName "kube-api-access-m2f67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:31.912169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.912148 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-xks-ca" pod "28f27cd2-47ab-4259-91c3-83a951f9d96f" (UID: "28f27cd2-47ab-4259-91c3-83a951f9d96f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:23:31.912461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:31.912437 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24956f45-acac-405b-b473-5850ab98e464-maas-api-xks-ca" pod "24956f45-acac-405b-b473-5850ab98e464" (UID: "24956f45-acac-405b-b473-5850ab98e464"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:23:32.010325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.010283 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24956f45-acac-405b-b473-5850ab98e464-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:32.010325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.010326 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwl8p\" (UniqueName: \"kubernetes.io/projected/28f27cd2-47ab-4259-91c3-83a951f9d96f-kube-api-access-dwl8p\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:32.010325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.010342 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:32.010671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.010444 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2f67\" (UniqueName: \"kubernetes.io/projected/24956f45-acac-405b-b473-5850ab98e464-kube-api-access-m2f67\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:32.010671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.010468 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24956f45-acac-405b-b473-5850ab98e464-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:32.010671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.010486 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f27cd2-47ab-4259-91c3-83a951f9d96f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:32.414360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.414265 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" Oct 31 13:23:32.414360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.414278 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn" event={"ID":"28f27cd2-47ab-4259-91c3-83a951f9d96f","Type":"ContainerDied","Data":"c53e1447f808db3527e6d50d6ed2b9fea384eb2003c6afa5692f5368c42164b6"} Oct 31 13:23:32.414360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.414337 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c53e1447f808db3527e6d50d6ed2b9fea384eb2003c6afa5692f5368c42164b6" Oct 31 13:23:32.416832 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.416806 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" event={"ID":"24956f45-acac-405b-b473-5850ab98e464","Type":"ContainerDied","Data":"c12d1a342674c77b01d672d078da1baa476d4a9064b6808a4dab85d6a4564eb4"} Oct 31 13:23:32.417029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.417009 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c12d1a342674c77b01d672d078da1baa476d4a9064b6808a4dab85d6a4564eb4" Oct 31 13:23:32.417214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:32.417039 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db" Oct 31 13:23:42.421835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:42.421773 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:23:42.893664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:42.890608 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5"] Oct 31 13:23:42.893664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:42.890892 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" containerID="cri-o://a99bd2295601afa7c89b6d17e4df36fe4462bdc649fc5ec4ce2eb0ba7379b012" gracePeriod=30 Oct 31 13:23:44.611022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.605386 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh"] Oct 31 13:23:44.617568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.606557 5074 cpu_manager.go:414] "containerMap: removing container" podUID="24956f45-acac-405b-b473-5850ab98e464" containerName="revoke-keys" Oct 31 13:23:44.617813 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.617791 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24956f45-acac-405b-b473-5850ab98e464" containerName="revoke-keys" Oct 31 13:23:44.618019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.618004 5074 cpu_manager.go:414] "containerMap: removing container" podUID="28f27cd2-47ab-4259-91c3-83a951f9d96f" containerName="revoke-keys" Oct 31 13:23:44.618117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.618106 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f27cd2-47ab-4259-91c3-83a951f9d96f" containerName="revoke-keys" Oct 31 13:23:44.618476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.618457 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="24956f45-acac-405b-b473-5850ab98e464" containerName="revoke-keys" Oct 31 13:23:44.618647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.618619 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f27cd2-47ab-4259-91c3-83a951f9d96f" containerName="revoke-keys" Oct 31 13:23:44.639753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.639072 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh"] Oct 31 13:23:44.639753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.639217 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.646256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.646220 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm"] Oct 31 13:23:44.654732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.654705 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm"] Oct 31 13:23:44.654876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.654791 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.680308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.680279 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5a9e45-openshift-default-dockercfg-szwmr\"" Oct 31 13:23:44.706452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.706418 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.706762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.706738 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784ac98e-380a-450d-834f-fa8bcd548b59-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.706886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.706798 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.706952 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.706919 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784ac98e-380a-450d-834f-fa8bcd548b59-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.707017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.706955 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.707017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.707006 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.707131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.707082 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.707352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.707334 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.707436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.707406 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrct\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-kube-api-access-qqrct\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.809372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809327 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.809581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809434 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75gr\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-kube-api-access-r75gr\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.809581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809466 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrct\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-kube-api-access-qqrct\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.809581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809498 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b2529d-286c-46e2-adf9-81367705d49a-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.809802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809754 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.809865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809796 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.809865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809839 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784ac98e-380a-450d-834f-fa8bcd548b59-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.810005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809890 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.810005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.809942 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b2529d-286c-46e2-adf9-81367705d49a-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.810116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810025 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.810116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810049 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.810361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810334 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784ac98e-380a-450d-834f-fa8bcd548b59-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.810465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810403 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.810524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810460 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.810583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810543 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.810583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810574 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.810743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810625 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.810743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810667 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784ac98e-380a-450d-834f-fa8bcd548b59-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.810743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810682 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.810743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810737 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.810945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.810922 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.811207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.811187 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.811324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.811229 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.813188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.813163 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.813670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.813626 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784ac98e-380a-450d-834f-fa8bcd548b59-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.829558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.829532 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrct\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-kube-api-access-qqrct\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.838004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.837970 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.912849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.912751 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75gr\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-kube-api-access-r75gr\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.912849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.912810 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b2529d-286c-46e2-adf9-81367705d49a-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.913096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.912942 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b2529d-286c-46e2-adf9-81367705d49a-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.913096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913013 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.913096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913036 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.913266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913159 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.913266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913203 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.913266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913235 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.913438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913268 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.914164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913786 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.914164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913878 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.914164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.913937 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.914164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.914119 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.914489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.914270 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b2529d-286c-46e2-adf9-81367705d49a-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.916074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.916050 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.916307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.916282 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b2529d-286c-46e2-adf9-81367705d49a-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.934250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.934223 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75gr\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-kube-api-access-r75gr\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.936477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.936444 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:44.970690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.963768 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:44.976694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:44.974561 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:45.778226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.778173 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm"] Oct 31 13:23:45.806485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.806429 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh"] Oct 31 13:23:45.859584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.857959 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:45.859584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.858041 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:45.859584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.858070 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:45.865297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.861329 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:45.865297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.861434 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:45.865297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:45.861493 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:46.399559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:46.399439 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" event={"ID":"784ac98e-380a-450d-834f-fa8bcd548b59","Type":"ContainerStarted","Data":"b84e883da0d87e2072e699eec406884188d3b09e11b098608ecd05f13a2dd90c"} Oct 31 13:23:46.415117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:46.415070 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" event={"ID":"75b2529d-286c-46e2-adf9-81367705d49a","Type":"ContainerStarted","Data":"fc9cf4bc8911b18b3436cad41cc81b4692ce12415dbecdbccc9cf1a0c2146934"} Oct 31 13:23:47.336555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.336499 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt"] Oct 31 13:23:47.342127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.342100 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:47.355546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.354838 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt"] Oct 31 13:23:47.359947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.359919 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4023ad-dockercfg-m44m7\"" Oct 31 13:23:47.434886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.430464 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5vx\" (UniqueName: \"kubernetes.io/projected/d772dc21-8fd6-43a0-ba9d-be8854db5729-kube-api-access-hn5vx\") pod \"payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:47.435247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.435218 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d772dc21-8fd6-43a0-ba9d-be8854db5729-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:47.444689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.444661 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4023ad\"" Oct 31 13:23:47.450374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.450339 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" event={"ID":"75b2529d-286c-46e2-adf9-81367705d49a","Type":"ContainerStarted","Data":"76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c"} Oct 31 13:23:47.455826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.455793 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" event={"ID":"784ac98e-380a-450d-834f-fa8bcd548b59","Type":"ContainerStarted","Data":"020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856"} Oct 31 13:23:47.540124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.540077 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5vx\" (UniqueName: \"kubernetes.io/projected/d772dc21-8fd6-43a0-ba9d-be8854db5729-kube-api-access-hn5vx\") pod \"payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:47.552923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.552878 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d772dc21-8fd6-43a0-ba9d-be8854db5729-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:47.593010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.565675 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d772dc21-8fd6-43a0-ba9d-be8854db5729-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:47.681133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.681064 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podStartSLOduration=3.681041806 podStartE2EDuration="3.681041806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:47.527682459 +0000 UTC m=+3121.705400444" watchObservedRunningTime="2026-10-31 13:23:47.681041806 +0000 UTC m=+3121.858759790" Oct 31 13:23:47.729659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.712066 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podStartSLOduration=3.712046866 podStartE2EDuration="3.712046866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:47.604188791 +0000 UTC m=+3121.781906775" watchObservedRunningTime="2026-10-31 13:23:47.712046866 +0000 UTC m=+3121.889764851" Oct 31 13:23:47.828653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.826388 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5vx\" (UniqueName: \"kubernetes.io/projected/d772dc21-8fd6-43a0-ba9d-be8854db5729-kube-api-access-hn5vx\") pod \"payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:47.961930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.961852 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:48.002654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.996114 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:23:48.002654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.996175 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:23:48.002654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.996450 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:48.002654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:47.996514 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:48.015660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.014891 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:23:48.015660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.014978 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Oct 31 13:23:48.566039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.565580 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt"] Oct 31 13:23:48.590160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.590122 5074 generic.go:356] "Generic (PLEG): container finished" podID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerID="a99bd2295601afa7c89b6d17e4df36fe4462bdc649fc5ec4ce2eb0ba7379b012" exitCode=0 Oct 31 13:23:48.628098 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.591582 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" event={"ID":"8ae025a9-1bf1-46b7-96c3-b0e0cd679388","Type":"ContainerDied","Data":"a99bd2295601afa7c89b6d17e4df36fe4462bdc649fc5ec4ce2eb0ba7379b012"} Oct 31 13:23:48.847470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.847443 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:23:49.002991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.976125 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:23:49.002991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.976184 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:23:49.002991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.976571 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:23:49.002991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:48.976602 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044146 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-podinfo\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044315 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-data\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044450 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-envoy\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044495 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-token\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-token\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044539 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istiod-ca-cert\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044573 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-kube-api-access-lb7pg\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-kube-api-access-lb7pg\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044656 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-certs\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044693 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-socket\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.045658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.044798 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-credential-socket\") pod \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\" (UID: \"8ae025a9-1bf1-46b7-96c3-b0e0cd679388\") " Oct 31 13:23:49.046441 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.045827 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-credential-socket" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:49.046441 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.046389 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-certs" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:49.047470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.047434 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-data" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:49.047695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.047664 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-socket" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:49.048094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.048076 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istiod-ca-cert" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:23:49.050681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.050604 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-envoy" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:23:49.055428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.055397 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-token" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:49.055831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.055805 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-podinfo" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:23:49.057559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.057518 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-kube-api-access-lb7pg" pod "8ae025a9-1bf1-46b7-96c3-b0e0cd679388" (UID: "8ae025a9-1bf1-46b7-96c3-b0e0cd679388"). InnerVolumeSpecName "kube-api-access-lb7pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147553 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147590 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147607 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147621 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147654 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147669 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147681 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147695 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.147836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.147709 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7pg\" (UniqueName: \"kubernetes.io/projected/8ae025a9-1bf1-46b7-96c3-b0e0cd679388-kube-api-access-lb7pg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:23:49.630939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.630561 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" event={"ID":"8ae025a9-1bf1-46b7-96c3-b0e0cd679388","Type":"ContainerDied","Data":"4e89d4d686dd39a6bc8989188037f735451888d83bbc6e62e0e9c421e998a985"} Oct 31 13:23:49.630939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.630578 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5" Oct 31 13:23:49.630939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.630621 5074 scope.go:122] "RemoveContainer" containerID="a99bd2295601afa7c89b6d17e4df36fe4462bdc649fc5ec4ce2eb0ba7379b012" Oct 31 13:23:49.633604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.633582 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" event={"ID":"d772dc21-8fd6-43a0-ba9d-be8854db5729","Type":"ContainerStarted","Data":"c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7"} Oct 31 13:23:49.633747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.633659 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" event={"ID":"d772dc21-8fd6-43a0-ba9d-be8854db5729","Type":"ContainerStarted","Data":"f1907255b2d44a16bed247b83869b852615b7328634d988379df8510f53ac176"} Oct 31 13:23:49.633814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.633753 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:23:49.662403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.660706 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" podStartSLOduration=2.660693943 podStartE2EDuration="2.660693943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:49.659030002 +0000 UTC m=+3123.836747987" watchObservedRunningTime="2026-10-31 13:23:49.660693943 +0000 UTC m=+3123.838411926" Oct 31 13:23:49.754304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.754256 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5"] Oct 31 13:23:49.759051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.759015 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-pqwx5"] Oct 31 13:23:49.961193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.961111 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" path="/var/lib/kubelet/pods/8ae025a9-1bf1-46b7-96c3-b0e0cd679388/volumes" Oct 31 13:23:50.006657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.990727 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:23:50.006657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.990793 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:23:50.006657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.991225 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:23:50.006657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:49.991263 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Oct 31 13:23:50.980236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:50.980196 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:23:50.983087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:50.983048 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:23:50.983613 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:50.980871 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:23:50.984048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:50.984017 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Oct 31 13:23:51.981208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:51.980747 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:23:51.981208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:51.980815 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Oct 31 13:23:51.990667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:51.984726 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:23:51.990667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:51.984782 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:23:52.779657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:52.770709 5074 scope.go:122] "RemoveContainer" containerID="a66a62a57fb8cc81b62eb87ed78487f335020459e00710f5280ca29ebbd8d24a" Oct 31 13:23:52.999544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:52.999506 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:23:53.000144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.000108 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:23:53.000685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.000662 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:23:53.008725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.008690 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Oct 31 13:23:53.056876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.056499 5074 scope.go:122] "RemoveContainer" containerID="9ff3f9a617d1f9493094483d058dd107f56e7ef03f6e2fb49f2872453877c1ae" Oct 31 13:23:53.636704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.585309 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh"] Oct 31 13:23:53.716660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.691562 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" Oct 31 13:23:53.716660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.691597 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" Oct 31 13:23:53.785414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.764860 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae025a9-1bf1-46b7-96c3-b0e0cd679388" containerName="istio-proxy" Oct 31 13:23:53.785414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.780483 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.839666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.812945 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5dbd39-openshift-default-dockercfg-l8k75\"" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927379 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927462 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927540 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8ad859-e258-4a60-b804-ab05271038b5-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927596 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927691 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927764 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927884 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8ad859-e258-4a60-b804-ab05271038b5-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927920 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.933878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.927952 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wb7\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-kube-api-access-66wb7\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:53.979911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.979876 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh"] Oct 31 13:23:53.980241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.980218 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body= Oct 31 13:23:53.980427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.980404 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" Oct 31 13:23:53.980715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.980687 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:23:53.980808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:53.980736 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:23:54.029657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.029605 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.029678 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.029736 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8ad859-e258-4a60-b804-ab05271038b5-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.029766 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.029811 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.029863 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.029962 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8ad859-e258-4a60-b804-ab05271038b5-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.030003 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.030026 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wb7\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-kube-api-access-66wb7\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.030155 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.030251 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.030422 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.030786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.030770 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.031261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.031217 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8ad859-e258-4a60-b804-ab05271038b5-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.033039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.033014 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.034208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.034184 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8ad859-e258-4a60-b804-ab05271038b5-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.095653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.094088 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wb7\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-kube-api-access-66wb7\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.118654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.117683 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.246418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.246388 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:54.600110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.600077 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh"] Oct 31 13:23:54.601506 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:23:54.601479 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8ad859_e258_4a60_b804_ab05271038b5.slice/crio-242be2c15a3b835422d4cd3c864b4f526a5f8d99354993aa2e413a6d996ea02a WatchSource:0}: Error finding container 242be2c15a3b835422d4cd3c864b4f526a5f8d99354993aa2e413a6d996ea02a: Status 404 returned error can't find the container with id 242be2c15a3b835422d4cd3c864b4f526a5f8d99354993aa2e413a6d996ea02a Oct 31 13:23:54.605191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.605157 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:54.605318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.605230 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:54.605318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.605259 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:54.904742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.904584 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" event={"ID":"cc8ad859-e258-4a60-b804-ab05271038b5","Type":"ContainerStarted","Data":"ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705"} Oct 31 13:23:54.904742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.904695 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" event={"ID":"cc8ad859-e258-4a60-b804-ab05271038b5","Type":"ContainerStarted","Data":"242be2c15a3b835422d4cd3c864b4f526a5f8d99354993aa2e413a6d996ea02a"} Oct 31 13:23:54.982659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.982514 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:54.982659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.982611 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:54.984735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.984708 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:54.985304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.985281 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:54.986318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.986003 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:23:54.986513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:54.986493 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:23:55.009703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:55.009650 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podStartSLOduration=2.009613836 podStartE2EDuration="2.009613836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:54.932701482 +0000 UTC m=+3129.110419465" watchObservedRunningTime="2026-10-31 13:23:55.009613836 +0000 UTC m=+3129.187331821" Oct 31 13:23:55.248858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:55.248823 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:55.250004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:55.249964 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:23:55.250161 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:55.250026 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:23:55.986017 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:23:55.985968 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:23:56.252666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:56.252583 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:56.930900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:56.930864 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:23:56.932509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:23:56.932473 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:24:00.685906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:00.685256 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:24:03.954974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:03.954931 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:24:10.028917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:10.028871 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g"] Oct 31 13:24:10.033682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:10.033655 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161ad445-78n7g"] Oct 31 13:24:11.952769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:11.952734 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba157151-94be-49d2-a2ba-7a0e84091f3a" path="/var/lib/kubelet/pods/ba157151-94be-49d2-a2ba-7a0e84091f3a/volumes" Oct 31 13:24:14.625315 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.625273 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq"] Oct 31 13:24:14.632017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.631991 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.651277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.651044 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq"] Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.717395 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.717496 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.717690 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.717734 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhls\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-kube-api-access-hrhls\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.717782 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.717925 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.717990 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.718090 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.744723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.718150 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820088 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820133 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhls\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-kube-api-access-hrhls\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820176 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820276 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820317 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820399 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820446 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820558 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820648 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820683 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.820984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.820958 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.821073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.821013 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.821073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.821061 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.821350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.821326 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.822867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.822839 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.823128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.823108 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.835548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.835522 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhls\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-kube-api-access-hrhls\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.835682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.835589 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:14.959478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:14.959396 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:15.261756 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:24:15.261710 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1aec07_a938_44ba_9a2d_e07385a1adf7.slice/crio-86788e1d614a6c8a9a467a7d2fe6a2c062696a1b02ef1f0f70072b921d38d788 WatchSource:0}: Error finding container 86788e1d614a6c8a9a467a7d2fe6a2c062696a1b02ef1f0f70072b921d38d788: Status 404 returned error can't find the container with id 86788e1d614a6c8a9a467a7d2fe6a2c062696a1b02ef1f0f70072b921d38d788 Oct 31 13:24:15.262499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.262471 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq"] Oct 31 13:24:15.265418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.265380 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.265532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.265462 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.265532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.265512 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:15.583956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.583844 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" event={"ID":"ec1aec07-a938-44ba-9a2d-e07385a1adf7","Type":"ContainerStarted","Data":"1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1"} Oct 31 13:24:15.583956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.583913 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" event={"ID":"ec1aec07-a938-44ba-9a2d-e07385a1adf7","Type":"ContainerStarted","Data":"86788e1d614a6c8a9a467a7d2fe6a2c062696a1b02ef1f0f70072b921d38d788"} Oct 31 13:24:15.617886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.617831 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" podStartSLOduration=1.617814611 podStartE2EDuration="1.617814611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:15.608248687 +0000 UTC m=+3149.785966672" watchObservedRunningTime="2026-10-31 13:24:15.617814611 +0000 UTC m=+3149.795532594" Oct 31 13:24:15.960497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:15.960420 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:16.962190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:16.962152 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:24:16.962588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:16.962210 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 13:24:17.960403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:17.960369 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:24:17.960592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:17.960426 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 13:24:18.960208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:18.960153 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:24:18.960671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:18.960229 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 13:24:19.960266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:19.960228 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:24:19.960713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:19.960284 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 13:24:20.072664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:20.071685 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq"] Oct 31 13:24:20.115614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:20.115200 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0069429-bdnsq"] Oct 31 13:24:20.971658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:20.967072 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 31 13:24:20.971658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:20.967140 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 31 13:24:21.961421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:21.961378 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b61e10-257e-4d14-a1f8-52488c352b34" path="/var/lib/kubelet/pods/63b61e10-257e-4d14-a1f8-52488c352b34/volumes" Oct 31 13:24:22.070443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:22.070398 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:22.072254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:22.072228 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:22.073976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:22.073952 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:24:25.986193 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:24:25.986155 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:24:29.644527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.644484 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx"] Oct 31 13:24:29.651547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.651514 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.666703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.666662 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx"] Oct 31 13:24:29.693691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.693589 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a453269b-dc26-4e66-833f-a84a512e0c10-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.693905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.693876 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.693977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.693928 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.694055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.694021 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a453269b-dc26-4e66-833f-a84a512e0c10-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.694107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.694073 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.694159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.694104 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.694294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.694275 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5dh\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-kube-api-access-rf5dh\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.694391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.694368 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.694481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.694466 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.795964 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a453269b-dc26-4e66-833f-a84a512e0c10-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796097 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796126 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796144 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a453269b-dc26-4e66-833f-a84a512e0c10-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796159 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796183 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796294 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5dh\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-kube-api-access-rf5dh\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796362 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796489 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796623 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-certs\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.796869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.796845 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a453269b-dc26-4e66-833f-a84a512e0c10-istiod-ca-cert\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.797139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.797113 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-data\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.797232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.797137 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.797232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.797216 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-credential-socket\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.798649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.798608 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-envoy\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.799210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.799187 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a453269b-dc26-4e66-833f-a84a512e0c10-istio-podinfo\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.808504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.808470 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-istio-token\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.808711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.808689 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5dh\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-kube-api-access-rf5dh\") pod \"e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:29.979755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:29.979716 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:30.303220 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:24:30.303184 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda453269b_dc26_4e66_833f_a84a512e0c10.slice/crio-55e0a657f5e9302840b86970c457bd1f18f8b62a1db4086151bd7b9639f18ee2 WatchSource:0}: Error finding container 55e0a657f5e9302840b86970c457bd1f18f8b62a1db4086151bd7b9639f18ee2: Status 404 returned error can't find the container with id 55e0a657f5e9302840b86970c457bd1f18f8b62a1db4086151bd7b9639f18ee2 Oct 31 13:24:30.304488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.304449 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx"] Oct 31 13:24:30.306955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.306918 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:30.307071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.307000 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:30.307137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.307069 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:30.800302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.800262 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" event={"ID":"a453269b-dc26-4e66-833f-a84a512e0c10","Type":"ContainerStarted","Data":"8a5df8eb03d44a6940bc33be6aff01e996d7e6c44d9c667f04bd97a508078e69"} Oct 31 13:24:30.800832 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.800327 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" event={"ID":"a453269b-dc26-4e66-833f-a84a512e0c10","Type":"ContainerStarted","Data":"55e0a657f5e9302840b86970c457bd1f18f8b62a1db4086151bd7b9639f18ee2"} Oct 31 13:24:30.826366 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.826309 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" podStartSLOduration=1.826293276 podStartE2EDuration="1.826293276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:30.824056089 +0000 UTC m=+3165.001774075" watchObservedRunningTime="2026-10-31 13:24:30.826293276 +0000 UTC m=+3165.004011259" Oct 31 13:24:30.980774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:30.980729 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:31.027588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:31.027546 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm"] Oct 31 13:24:31.031847 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:31.031815 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1d5a0d2-ddrpm"] Oct 31 13:24:31.993870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:31.993830 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 13:24:32.009776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:32.009725 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 13:24:32.021659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:32.020812 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b91eeea-72bc-475a-8671-a8da81a6a1e5" path="/var/lib/kubelet/pods/4b91eeea-72bc-475a-8671-a8da81a6a1e5/volumes" Oct 31 13:24:32.981091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:32.981049 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 13:24:32.983422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:32.983386 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 13:24:33.997657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:33.994093 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 13:24:33.997657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:33.994168 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 13:24:34.981069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:34.981024 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 13:24:34.982237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:34.982202 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 13:24:35.994662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:35.990749 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 13:24:35.994662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:35.990815 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 13:24:37.089257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:37.089202 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:37.090705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:37.090676 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:37.093050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:37.093029 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:24:42.849311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.849268 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq"] Oct 31 13:24:42.856860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.856837 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5"] Oct 31 13:24:42.857004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.856956 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:42.862550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.862510 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:42.862812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.862788 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c6c445-dockercfg-q9bv2\"" Oct 31 13:24:42.862942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.862748 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c6c445\"" Oct 31 13:24:42.870045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.869966 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq"] Oct 31 13:24:42.879658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.879137 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5"] Oct 31 13:24:43.021113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.988020 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:43.021113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.988260 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1285e99f-1285-4dd7-acf8-ba979766bc74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:43.021113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.988347 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hcqm\" (UniqueName: \"kubernetes.io/projected/1285e99f-1285-4dd7-acf8-ba979766bc74-kube-api-access-7hcqm\") pod \"payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:43.021113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:42.988608 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkk4s\" (UniqueName: \"kubernetes.io/projected/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-kube-api-access-nkk4s\") pod \"payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:43.096689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.093155 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkk4s\" (UniqueName: \"kubernetes.io/projected/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-kube-api-access-nkk4s\") pod \"payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:43.096689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.093286 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:43.096689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.093466 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1285e99f-1285-4dd7-acf8-ba979766bc74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:43.096689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.093556 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hcqm\" (UniqueName: \"kubernetes.io/projected/1285e99f-1285-4dd7-acf8-ba979766bc74-kube-api-access-7hcqm\") pod \"payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:43.148712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.148331 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:43.168675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.165425 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1285e99f-1285-4dd7-acf8-ba979766bc74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:43.224672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.223752 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkk4s\" (UniqueName: \"kubernetes.io/projected/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-kube-api-access-nkk4s\") pod \"payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:43.269653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.241370 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hcqm\" (UniqueName: \"kubernetes.io/projected/1285e99f-1285-4dd7-acf8-ba979766bc74-kube-api-access-7hcqm\") pod \"payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:43.488926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.488884 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:43.506371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.506337 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:43.809610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.797683 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq"] Oct 31 13:24:43.809610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:43.806748 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5"] Oct 31 13:24:43.820301 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:24:43.820269 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1285e99f_1285_4dd7_acf8_ba979766bc74.slice/crio-a2313fc6319c54e3d99743d225c6c70e9dc1a81c5e160719e4423e25d2829e8c WatchSource:0}: Error finding container a2313fc6319c54e3d99743d225c6c70e9dc1a81c5e160719e4423e25d2829e8c: Status 404 returned error can't find the container with id a2313fc6319c54e3d99743d225c6c70e9dc1a81c5e160719e4423e25d2829e8c Oct 31 13:24:44.027945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.027566 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb"] Oct 31 13:24:44.032788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.032759 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx"] Oct 31 13:24:44.036944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.036923 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c557a6-frbnb"] Oct 31 13:24:44.040773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.040751 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d652d47-dqqfx"] Oct 31 13:24:44.151804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.151713 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" event={"ID":"1285e99f-1285-4dd7-acf8-ba979766bc74","Type":"ContainerStarted","Data":"2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122"} Oct 31 13:24:44.151804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.151773 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" event={"ID":"1285e99f-1285-4dd7-acf8-ba979766bc74","Type":"ContainerStarted","Data":"a2313fc6319c54e3d99743d225c6c70e9dc1a81c5e160719e4423e25d2829e8c"} Oct 31 13:24:44.151804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.151800 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:44.153567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.153537 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" event={"ID":"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b","Type":"ContainerStarted","Data":"8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507"} Oct 31 13:24:44.153714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.153588 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:44.153714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.153606 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" event={"ID":"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b","Type":"ContainerStarted","Data":"b3091e09d4d6e111547dfcbbf436cb116d33e22e49026afc92c17b1742df219c"} Oct 31 13:24:44.175886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.175837 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" podStartSLOduration=2.17582457 podStartE2EDuration="2.17582457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:44.170120751 +0000 UTC m=+3178.347838743" watchObservedRunningTime="2026-10-31 13:24:44.17582457 +0000 UTC m=+3178.353542553" Oct 31 13:24:44.197235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:44.197177 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" podStartSLOduration=2.197158198 podStartE2EDuration="2.197158198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:44.192549269 +0000 UTC m=+3178.370267253" watchObservedRunningTime="2026-10-31 13:24:44.197158198 +0000 UTC m=+3178.374876186" Oct 31 13:24:45.943691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:45.943654 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca25edb-a97b-400c-b8fa-6cc2efdc6575" path="/var/lib/kubelet/pods/1ca25edb-a97b-400c-b8fa-6cc2efdc6575/volumes" Oct 31 13:24:45.944325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:45.944308 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53669b16-6aa1-4440-a00b-79156a01fcb9" path="/var/lib/kubelet/pods/53669b16-6aa1-4440-a00b-79156a01fcb9/volumes" Oct 31 13:24:53.452823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:53.452789 5074 scope.go:122] "RemoveContainer" containerID="1320d6377af3c1247f37df3404a38de9d9583dc9e40dad88377e4fa970c05904" Oct 31 13:24:53.464228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:53.464203 5074 scope.go:122] "RemoveContainer" containerID="fdff75e16c9bec78182094babaa0a45ebacc4f9ba13aae0e5b6b798d6e715ccb" Oct 31 13:24:53.474711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:53.474684 5074 scope.go:122] "RemoveContainer" containerID="17cf5b0f3a014e8d72b299c8d8804cc234336ef9829c72d05178648e0a672b9c" Oct 31 13:24:53.484778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:53.484749 5074 scope.go:122] "RemoveContainer" containerID="0676c7b06938c8e6211059d8c1ea79c45b0c6110c25671659b0b7144f9b3c0da" Oct 31 13:24:55.036393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:55.036332 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc"] Oct 31 13:24:55.041016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:55.040986 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0f652-26ljc"] Oct 31 13:24:55.255324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:55.255290 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:24:55.255583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:55.255395 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:24:55.944850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:24:55.944817 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb8e7cb-2a7e-4506-858e-1c4689817e03" path="/var/lib/kubelet/pods/dcb8e7cb-2a7e-4506-858e-1c4689817e03/volumes" Oct 31 13:24:55.984229 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:24:55.984195 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:25:05.037599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:05.037551 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4"] Oct 31 13:25:05.042570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:05.042535 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b4f77c-zwgw4"] Oct 31 13:25:05.942512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:05.942475 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2716d1-f401-47c2-a7e8-cfe56628a063" path="/var/lib/kubelet/pods/4a2716d1-f401-47c2-a7e8-cfe56628a063/volumes" Oct 31 13:25:07.639285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:07.639184 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:25:12.004043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.004007 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z"] Oct 31 13:25:12.009025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.009000 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.013206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.013175 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:25:12.015182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.014428 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:25:12.015182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.014539 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:25:12.015182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.014704 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:25:12.017980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.017956 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z"] Oct 31 13:25:12.065139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.065107 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.065302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.065245 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.065302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.065286 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6ph\" (UniqueName: \"kubernetes.io/projected/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-kube-api-access-qh6ph\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.166332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.166299 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.166527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.166345 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6ph\" (UniqueName: \"kubernetes.io/projected/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-kube-api-access-qh6ph\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.166615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.166564 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.167263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.167241 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.168965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.168937 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.179492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.179463 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6ph\" (UniqueName: \"kubernetes.io/projected/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-kube-api-access-qh6ph\") pod \"maas-api-revoke-sub-aeaf5726-brz8z\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.338723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.327356 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:12.707497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:12.707121 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z"] Oct 31 13:25:13.005411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:13.005364 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" event={"ID":"cac975c0-e17a-4818-bbc0-8dc9e4702c4c","Type":"ContainerStarted","Data":"6f3c9523623abf8bfed697f01fb90a4667d00bcc4c1b69b3823e7e03b95522c8"} Oct 31 13:25:14.029167 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:14.029064 5074 generic.go:356] "Generic (PLEG): container finished" podID="cac975c0-e17a-4818-bbc0-8dc9e4702c4c" containerID="4f4e38bae3cfa66e2ac8fcf10ff42f54241d62f1ff4ff542820940e67809915d" exitCode=0 Oct 31 13:25:14.029167 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:14.029138 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" event={"ID":"cac975c0-e17a-4818-bbc0-8dc9e4702c4c","Type":"ContainerDied","Data":"4f4e38bae3cfa66e2ac8fcf10ff42f54241d62f1ff4ff542820940e67809915d"} Oct 31 13:25:15.242779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.242752 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:15.327246 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.327142 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-xks-ca\") pod \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " Oct 31 13:25:15.327453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.327432 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-openshift-ca\") pod \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " Oct 31 13:25:15.327531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.327503 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-kube-api-access-qh6ph\" (UniqueName: \"kubernetes.io/projected/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-kube-api-access-qh6ph\") pod \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\" (UID: \"cac975c0-e17a-4818-bbc0-8dc9e4702c4c\") " Oct 31 13:25:15.327849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.327821 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-openshift-ca" pod "cac975c0-e17a-4818-bbc0-8dc9e4702c4c" (UID: "cac975c0-e17a-4818-bbc0-8dc9e4702c4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:25:15.328215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.328196 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:15.329539 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.329510 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-xks-ca" pod "cac975c0-e17a-4818-bbc0-8dc9e4702c4c" (UID: "cac975c0-e17a-4818-bbc0-8dc9e4702c4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:25:15.330250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.330227 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-kube-api-access-qh6ph" pod "cac975c0-e17a-4818-bbc0-8dc9e4702c4c" (UID: "cac975c0-e17a-4818-bbc0-8dc9e4702c4c"). InnerVolumeSpecName "kube-api-access-qh6ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:15.429434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.429393 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6ph\" (UniqueName: \"kubernetes.io/projected/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-kube-api-access-qh6ph\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:15.429434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:15.429432 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac975c0-e17a-4818-bbc0-8dc9e4702c4c-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:16.050794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:16.050746 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" Oct 31 13:25:16.050794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:16.050780 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z" event={"ID":"cac975c0-e17a-4818-bbc0-8dc9e4702c4c","Type":"ContainerDied","Data":"6f3c9523623abf8bfed697f01fb90a4667d00bcc4c1b69b3823e7e03b95522c8"} Oct 31 13:25:16.051072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:16.050830 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f3c9523623abf8bfed697f01fb90a4667d00bcc4c1b69b3823e7e03b95522c8" Oct 31 13:25:23.142417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.142376 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99"] Oct 31 13:25:23.695679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.695616 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p"] Oct 31 13:25:23.696924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.696892 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cac975c0-e17a-4818-bbc0-8dc9e4702c4c" containerName="revoke-keys" Oct 31 13:25:23.697107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.697092 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac975c0-e17a-4818-bbc0-8dc9e4702c4c" containerName="revoke-keys" Oct 31 13:25:23.697453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.697433 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac975c0-e17a-4818-bbc0-8dc9e4702c4c" containerName="revoke-keys" Oct 31 13:25:23.717012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.716980 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p"] Oct 31 13:25:23.717464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.717348 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.726450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.726337 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:25:23.728422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.728398 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:25:23.728568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.728441 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:25:23.729000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.728980 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:25:23.823695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.821904 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4zj\" (UniqueName: \"kubernetes.io/projected/91527f14-f515-4b3e-aa72-55bc824113a9-kube-api-access-9g4zj\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.823695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.822209 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.823695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.822273 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.923765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.923721 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.923952 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.923818 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.923952 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.923945 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4zj\" (UniqueName: \"kubernetes.io/projected/91527f14-f515-4b3e-aa72-55bc824113a9-kube-api-access-9g4zj\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.924680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.924655 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.926851 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.926823 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:23.941877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:23.941844 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4zj\" (UniqueName: \"kubernetes.io/projected/91527f14-f515-4b3e-aa72-55bc824113a9-kube-api-access-9g4zj\") pod \"maas-api-revoke-sub-54d9a7a5-m544p\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:24.052359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:24.052317 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:24.355711 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:25:24.355596 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91527f14_f515_4b3e_aa72_55bc824113a9.slice/crio-5f25569960af215968963bbf2188e74db4323d053b8820d793fa16ba3d060051 WatchSource:0}: Error finding container 5f25569960af215968963bbf2188e74db4323d053b8820d793fa16ba3d060051: Status 404 returned error can't find the container with id 5f25569960af215968963bbf2188e74db4323d053b8820d793fa16ba3d060051 Oct 31 13:25:24.356277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:24.356252 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p"] Oct 31 13:25:25.177944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:25.177908 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" event={"ID":"91527f14-f515-4b3e-aa72-55bc824113a9","Type":"ContainerStarted","Data":"f71185d810d3966273f0b7598d92eddc8407a279cc37d9b4aa477d158e8c75ac"} Oct 31 13:25:25.178385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:25.177963 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" event={"ID":"91527f14-f515-4b3e-aa72-55bc824113a9","Type":"ContainerStarted","Data":"5f25569960af215968963bbf2188e74db4323d053b8820d793fa16ba3d060051"} Oct 31 13:25:25.199132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:25.199073 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" podStartSLOduration=2.199059959 podStartE2EDuration="2.199059959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:25.197102051 +0000 UTC m=+3219.374820036" watchObservedRunningTime="2026-10-31 13:25:25.199059959 +0000 UTC m=+3219.376777943" Oct 31 13:25:25.994021 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:25:25.993967 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:25:26.189941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:26.189903 5074 generic.go:356] "Generic (PLEG): container finished" podID="91527f14-f515-4b3e-aa72-55bc824113a9" containerID="f71185d810d3966273f0b7598d92eddc8407a279cc37d9b4aa477d158e8c75ac" exitCode=0 Oct 31 13:25:26.190133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:26.190008 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" event={"ID":"91527f14-f515-4b3e-aa72-55bc824113a9","Type":"ContainerDied","Data":"f71185d810d3966273f0b7598d92eddc8407a279cc37d9b4aa477d158e8c75ac"} Oct 31 13:25:27.393166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.393134 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:27.411324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.411289 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91527f14-f515-4b3e-aa72-55bc824113a9-kube-api-access-9g4zj\" (UniqueName: \"kubernetes.io/projected/91527f14-f515-4b3e-aa72-55bc824113a9-kube-api-access-9g4zj\") pod \"91527f14-f515-4b3e-aa72-55bc824113a9\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " Oct 31 13:25:27.411472 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.411403 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-xks-ca\") pod \"91527f14-f515-4b3e-aa72-55bc824113a9\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " Oct 31 13:25:27.411621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.411600 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-openshift-ca\") pod \"91527f14-f515-4b3e-aa72-55bc824113a9\" (UID: \"91527f14-f515-4b3e-aa72-55bc824113a9\") " Oct 31 13:25:27.412845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.412815 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-openshift-ca" pod "91527f14-f515-4b3e-aa72-55bc824113a9" (UID: "91527f14-f515-4b3e-aa72-55bc824113a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:25:27.419774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.414745 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91527f14-f515-4b3e-aa72-55bc824113a9-kube-api-access-9g4zj" pod "91527f14-f515-4b3e-aa72-55bc824113a9" (UID: "91527f14-f515-4b3e-aa72-55bc824113a9"). InnerVolumeSpecName "kube-api-access-9g4zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:27.419774 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.416087 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-xks-ca" pod "91527f14-f515-4b3e-aa72-55bc824113a9" (UID: "91527f14-f515-4b3e-aa72-55bc824113a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:25:27.514346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.514312 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4zj\" (UniqueName: \"kubernetes.io/projected/91527f14-f515-4b3e-aa72-55bc824113a9-kube-api-access-9g4zj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:27.514346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.514342 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:27.514553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:27.514354 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91527f14-f515-4b3e-aa72-55bc824113a9-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:28.155891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.148987 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" podUID="199d1ca7-a6fc-43ce-9392-216fdef9f347" containerName="payload-processing" containerID="cri-o://7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2" gracePeriod=25 Oct 31 13:25:28.210845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.210774 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" Oct 31 13:25:28.210845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.210797 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p" event={"ID":"91527f14-f515-4b3e-aa72-55bc824113a9","Type":"ContainerDied","Data":"5f25569960af215968963bbf2188e74db4323d053b8820d793fa16ba3d060051"} Oct 31 13:25:28.211047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.210865 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f25569960af215968963bbf2188e74db4323d053b8820d793fa16ba3d060051" Oct 31 13:25:28.281128 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:25:28.281099 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199d1ca7_a6fc_43ce_9392_216fdef9f347.slice/crio-conmon-7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:25:28.557713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.557678 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:25:28.659891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.659859 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-kube-api-access-7dprb\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-kube-api-access-7dprb\") pod \"199d1ca7-a6fc-43ce-9392-216fdef9f347\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " Oct 31 13:25:28.660101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.659906 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-provider-credentials\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-provider-credentials\") pod \"199d1ca7-a6fc-43ce-9392-216fdef9f347\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " Oct 31 13:25:28.660101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.660073 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-routing-overlay-volume\") pod \"199d1ca7-a6fc-43ce-9392-216fdef9f347\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " Oct 31 13:25:28.660252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.660143 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-plugins-config-volume\") pod \"199d1ca7-a6fc-43ce-9392-216fdef9f347\" (UID: \"199d1ca7-a6fc-43ce-9392-216fdef9f347\") " Oct 31 13:25:28.660718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.660497 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-routing-overlay-volume" pod "199d1ca7-a6fc-43ce-9392-216fdef9f347" (UID: "199d1ca7-a6fc-43ce-9392-216fdef9f347"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:25:28.660894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.660796 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-plugins-config-volume" pod "199d1ca7-a6fc-43ce-9392-216fdef9f347" (UID: "199d1ca7-a6fc-43ce-9392-216fdef9f347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:25:28.661427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.661390 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:28.661557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.661429 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199d1ca7-a6fc-43ce-9392-216fdef9f347-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:28.662408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.662380 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-provider-credentials" pod "199d1ca7-a6fc-43ce-9392-216fdef9f347" (UID: "199d1ca7-a6fc-43ce-9392-216fdef9f347"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:28.662408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.662402 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-kube-api-access-7dprb" pod "199d1ca7-a6fc-43ce-9392-216fdef9f347" (UID: "199d1ca7-a6fc-43ce-9392-216fdef9f347"). InnerVolumeSpecName "kube-api-access-7dprb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:25:28.763121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.763083 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dprb\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-kube-api-access-7dprb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:28.763121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:28.763118 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/199d1ca7-a6fc-43ce-9392-216fdef9f347-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:25:29.303588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.303548 5074 generic.go:356] "Generic (PLEG): container finished" podID="199d1ca7-a6fc-43ce-9392-216fdef9f347" containerID="7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2" exitCode=0 Oct 31 13:25:29.303791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.303596 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" Oct 31 13:25:29.303946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.303919 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" event={"ID":"199d1ca7-a6fc-43ce-9392-216fdef9f347","Type":"ContainerDied","Data":"7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2"} Oct 31 13:25:29.304016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.303979 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99" event={"ID":"199d1ca7-a6fc-43ce-9392-216fdef9f347","Type":"ContainerDied","Data":"042bd8ee8e9050c932bec3f8430774eb998015c6b95ef8d6a1b5d5ff596c1c49"} Oct 31 13:25:29.304016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.304007 5074 scope.go:122] "RemoveContainer" containerID="7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2" Oct 31 13:25:29.318410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.318264 5074 scope.go:122] "RemoveContainer" containerID="7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2" Oct 31 13:25:29.318718 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:25:29.318624 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2\": container with ID starting with 7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2 not found: ID does not exist" containerID="7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2" Oct 31 13:25:29.318718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.318683 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2"} err="failed to get container status \"7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2\": rpc error: code = NotFound desc = could not find container \"7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2\": container with ID starting with 7343e9fecc29371201e3ebcb7dce63fb8669e3e6fa0020cef4bb8670957c47a2 not found: ID does not exist" Oct 31 13:25:29.338999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.338969 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99"] Oct 31 13:25:29.344093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.344063 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-6c58c9n99"] Oct 31 13:25:29.946147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:29.946112 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199d1ca7-a6fc-43ce-9392-216fdef9f347" path="/var/lib/kubelet/pods/199d1ca7-a6fc-43ce-9392-216fdef9f347/volumes" Oct 31 13:25:45.981257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.981218 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws"] Oct 31 13:25:45.983076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.983046 5074 cpu_manager.go:414] "containerMap: removing container" podUID="199d1ca7-a6fc-43ce-9392-216fdef9f347" containerName="payload-processing" Oct 31 13:25:45.983249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.983230 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199d1ca7-a6fc-43ce-9392-216fdef9f347" containerName="payload-processing" Oct 31 13:25:45.983383 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.983369 5074 cpu_manager.go:414] "containerMap: removing container" podUID="91527f14-f515-4b3e-aa72-55bc824113a9" containerName="revoke-keys" Oct 31 13:25:45.983498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.983483 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91527f14-f515-4b3e-aa72-55bc824113a9" containerName="revoke-keys" Oct 31 13:25:45.983824 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.983804 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="199d1ca7-a6fc-43ce-9392-216fdef9f347" containerName="payload-processing" Oct 31 13:25:45.983961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.983945 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="91527f14-f515-4b3e-aa72-55bc824113a9" containerName="revoke-keys" Oct 31 13:25:45.990406 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.990386 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:45.999980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:45.999952 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"routing-overlay\"" Oct 31 13:25:46.000515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.000492 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"openshift-service-ca.crt\"" Oct 31 13:25:46.000736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.000703 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"kube-root-ca.crt\"" Oct 31 13:25:46.000829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.000804 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"payload-processing-external-model-e2e--527e58cd-dockercfg-z6zpw\"" Oct 31 13:25:46.000903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.000864 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5a9e45\"" Oct 31 13:25:46.005425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.005400 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws"] Oct 31 13:25:46.050134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.050092 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.050318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.050268 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.050439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.050418 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztwjh\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-kube-api-access-ztwjh\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.050665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.050645 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.099189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.099155 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"agc-cred-611247b9727e\"" Oct 31 13:25:46.152343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.152306 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztwjh\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-kube-api-access-ztwjh\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.152507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.152477 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.152585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.152571 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.152727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.152705 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.155114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.155083 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.158833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.158801 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"routing-overlay\"" Oct 31 13:25:46.158833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.158801 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5a9e45\"" Oct 31 13:25:46.203672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.163502 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.203672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.163889 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.203672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.170384 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"kube-root-ca.crt\"" Oct 31 13:25:46.203672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.183607 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"openshift-service-ca.crt\"" Oct 31 13:25:46.235020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.232745 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztwjh\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-kube-api-access-ztwjh\") pod \"payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:46.341653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.331008 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5a9e45\"/\"payload-processing-external-model-e2e--527e58cd-dockercfg-z6zpw\"" Oct 31 13:25:46.341653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:46.332387 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:47.021707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:47.021063 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws"] Oct 31 13:25:47.860665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:47.857248 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" event={"ID":"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11","Type":"ContainerStarted","Data":"17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde"} Oct 31 13:25:47.860665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:47.857317 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" event={"ID":"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11","Type":"ContainerStarted","Data":"0a374173aa4ddb3154d8ffef203bdb3978e4f0d8c76fce405d0d74174d4d5f4d"} Oct 31 13:25:47.872664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:47.869708 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:25:47.935657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:47.922450 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" podStartSLOduration=2.922426705 podStartE2EDuration="2.922426705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:47.90214455 +0000 UTC m=+3242.079862535" watchObservedRunningTime="2026-10-31 13:25:47.922426705 +0000 UTC m=+3242.100144693" Oct 31 13:25:49.057794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:49.057755 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r"] Oct 31 13:25:49.067271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:49.067242 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd25297-jc92r"] Oct 31 13:25:49.955332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:49.955289 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c67e78-4fd5-4241-836a-ccabf8cd3c96" path="/var/lib/kubelet/pods/11c67e78-4fd5-4241-836a-ccabf8cd3c96/volumes" Oct 31 13:25:53.647656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:53.622544 5074 scope.go:122] "RemoveContainer" containerID="aefb70e0fe79bac6a84c6db1a3bbd9d15daa3992b60e79c5414eae73fd5fda25" Oct 31 13:25:53.745151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:53.745102 5074 scope.go:122] "RemoveContainer" containerID="b6e4850ba1f5b3cd9b76d88f3f3f2dc21e154ca093e89e9ca1ff9b92842aa7ee" Oct 31 13:25:54.000963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.000915 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:25:54.055615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.055578 5074 scope.go:122] "RemoveContainer" containerID="da7c2c20958615fd0104f303ae9aa41cfda3e151e7047d77c65455d87c28243c" Oct 31 13:25:54.096835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.096791 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:25:54.285010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.249772 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:25:54.285010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.250170 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 13:25:54.285010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.274939 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 31 13:25:54.285010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.284189 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:25:54.314663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.306550 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:25:54.416483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.416434 5074 scope.go:122] "RemoveContainer" containerID="0ca2a03399bbce046b5476b1343115c602aa474679f4b88ec92aafb54255a6a1" Oct 31 13:25:54.447684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.447652 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:25:54.472064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.472038 5074 scope.go:122] "RemoveContainer" containerID="a07012d460debad3f1951cee9611fb7166e8fa42cc88d76e556aba068033c31c" Oct 31 13:25:54.478599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.478574 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2"] Oct 31 13:25:54.492028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.491436 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2"] Oct 31 13:25:54.492028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.491598 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.493677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.492989 5074 scope.go:122] "RemoveContainer" containerID="b89471695e5b593e997f5367ed1558c07c29d0e4cb6b9beccae450ba37fc2430" Oct 31 13:25:54.502449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.502426 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2"] Oct 31 13:25:54.516777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.513837 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.521068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.521041 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-e2e--64249880-dockercfg-l2zw7\"" Oct 31 13:25:54.521175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.521108 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"agc-cred-7d5d1d5a1265\"" Oct 31 13:25:54.521257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.521238 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"agc-cred-b40702efee3d\"" Oct 31 13:25:54.526680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.526655 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-842ac9\"" Oct 31 13:25:54.530602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.530582 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"openshift-service-ca.crt\"" Oct 31 13:25:54.542681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.542566 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2"] Oct 31 13:25:54.559587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.559343 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-15a6a2\"" Oct 31 13:25:54.559587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.559463 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"routing-overlay\"" Oct 31 13:25:54.559587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.559531 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"openshift-service-ca.crt\"" Oct 31 13:25:54.559890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.559651 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"kube-root-ca.crt\"" Oct 31 13:25:54.559890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.559709 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-842ac9\"/\"routing-overlay\"" Oct 31 13:25:54.559890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.559846 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"kube-root-ca.crt\"" Oct 31 13:25:54.619731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.619690 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-3a2aaf478b2f\"" Oct 31 13:25:54.619731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.619712 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"payload-processing-external-model-e2e--7a720dc0-dockercfg-592fk\"" Oct 31 13:25:54.619731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.619692 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-15a6a2\"/\"agc-cred-e2cf7058e6c2\"" Oct 31 13:25:54.622987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.622956 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.623143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.623006 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.623143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.623038 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.623242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.623211 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.623344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.623324 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.623423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.623408 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.623500 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.623487 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cllg5\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-kube-api-access-cllg5\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.623538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.623513 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8v9n\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-kube-api-access-r8v9n\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726374 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726444 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726486 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726552 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726660 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726734 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726823 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cllg5\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-kube-api-access-cllg5\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.726915 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8v9n\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-kube-api-access-r8v9n\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.728563 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.731458 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.733117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.732125 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.734323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.733144 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.767168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.761223 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.767168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.762498 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.772703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.772673 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cllg5\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-kube-api-access-cllg5\") pod \"payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.777908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.777288 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8v9n\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-kube-api-access-r8v9n\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:54.823555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.823461 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:54.833431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:54.833399 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:55.378843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:55.378795 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2"] Oct 31 13:25:55.379542 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:25:55.379505 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1530f044_02c1_47f5_8daf_43e622687a0c.slice/crio-e3fa8339cf94dd8fa44a7d342e06a49c570c2558df40c5dd1f5f181e8fb4ed7b WatchSource:0}: Error finding container e3fa8339cf94dd8fa44a7d342e06a49c570c2558df40c5dd1f5f181e8fb4ed7b: Status 404 returned error can't find the container with id e3fa8339cf94dd8fa44a7d342e06a49c570c2558df40c5dd1f5f181e8fb4ed7b Oct 31 13:25:55.387856 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:25:55.387832 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95d4cfd_3c3f_44d5_bb9f_39536dd3f6dc.slice/crio-286be924aa61e7746d22bf75c32fdd33cc170d4b555fec94bd0c7b4061f6cc2c WatchSource:0}: Error finding container 286be924aa61e7746d22bf75c32fdd33cc170d4b555fec94bd0c7b4061f6cc2c: Status 404 returned error can't find the container with id 286be924aa61e7746d22bf75c32fdd33cc170d4b555fec94bd0c7b4061f6cc2c Oct 31 13:25:55.391019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:55.390995 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2"] Oct 31 13:25:55.460316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:55.460284 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" event={"ID":"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc","Type":"ContainerStarted","Data":"286be924aa61e7746d22bf75c32fdd33cc170d4b555fec94bd0c7b4061f6cc2c"} Oct 31 13:25:55.461608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:55.461581 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" event={"ID":"1530f044-02c1-47f5-8daf-43e622687a0c","Type":"ContainerStarted","Data":"e3fa8339cf94dd8fa44a7d342e06a49c570c2558df40c5dd1f5f181e8fb4ed7b"} Oct 31 13:25:55.992005 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:25:55.991959 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:25:56.476798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:56.476757 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" event={"ID":"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc","Type":"ContainerStarted","Data":"2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5"} Oct 31 13:25:56.477015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:56.476836 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:25:56.479279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:56.479243 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" event={"ID":"1530f044-02c1-47f5-8daf-43e622687a0c","Type":"ContainerStarted","Data":"8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329"} Oct 31 13:25:56.479561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:56.479522 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:25:56.561057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:56.560993 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" podStartSLOduration=2.560974038 podStartE2EDuration="2.560974038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:56.554296752 +0000 UTC m=+3250.732014736" watchObservedRunningTime="2026-10-31 13:25:56.560974038 +0000 UTC m=+3250.738692023" Oct 31 13:25:56.612933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:56.612867 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" podStartSLOduration=2.6128518339999998 podStartE2EDuration="2.612851834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:56.608563093 +0000 UTC m=+3250.786281095" watchObservedRunningTime="2026-10-31 13:25:56.612851834 +0000 UTC m=+3250.790569842" Oct 31 13:25:59.885130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:25:59.885092 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:26:07.493234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:07.491931 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:26:07.493234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:07.492037 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:26:12.097990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:12.097953 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm"] Oct 31 13:26:12.106222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:12.106184 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20705af-529sm"] Oct 31 13:26:13.311296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:13.311262 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:26:13.941233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:13.941201 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b022f1-d4f7-47a8-b086-3b8c53b85ebb" path="/var/lib/kubelet/pods/f2b022f1-d4f7-47a8-b086-3b8c53b85ebb/volumes" Oct 31 13:26:14.031023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:14.030989 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj"] Oct 31 13:26:14.035629 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:14.035600 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86e5c27-ls8jj"] Oct 31 13:26:16.033860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:16.033813 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf" path="/var/lib/kubelet/pods/9f4ab14a-fdcd-4f0a-afcf-60b51756dfdf/volumes" Oct 31 13:26:16.434750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:16.430757 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:26:19.478850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.478808 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5"] Oct 31 13:26:19.484278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.484252 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.488873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.488852 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:26:19.489176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.489140 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:26:19.490133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.490113 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:26:19.490235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.490160 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:26:19.494235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.494212 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5"] Oct 31 13:26:19.676026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.675982 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lx75\" (UniqueName: \"kubernetes.io/projected/a7abb12e-61a4-42bc-ae37-e6120396e413-kube-api-access-6lx75\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.676659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.676605 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.676947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.676925 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.778883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.778777 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.778883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.778855 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lx75\" (UniqueName: \"kubernetes.io/projected/a7abb12e-61a4-42bc-ae37-e6120396e413-kube-api-access-6lx75\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.779210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.779183 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.779969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.779937 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.781450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.781427 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.792272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.792249 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lx75\" (UniqueName: \"kubernetes.io/projected/a7abb12e-61a4-42bc-ae37-e6120396e413-kube-api-access-6lx75\") pod \"maas-api-revoke-sub-247da670-j9nw5\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:19.804820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:19.804787 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:20.101706 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:26:20.101667 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7abb12e_61a4_42bc_ae37_e6120396e413.slice/crio-393948bd51fba651abadb1d9aa44896431990aaa2d93c3c67e1bd85f53fcae5e WatchSource:0}: Error finding container 393948bd51fba651abadb1d9aa44896431990aaa2d93c3c67e1bd85f53fcae5e: Status 404 returned error can't find the container with id 393948bd51fba651abadb1d9aa44896431990aaa2d93c3c67e1bd85f53fcae5e Oct 31 13:26:20.102826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:20.102798 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5"] Oct 31 13:26:21.007550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:21.007501 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" event={"ID":"a7abb12e-61a4-42bc-ae37-e6120396e413","Type":"ContainerStarted","Data":"d8751971e63afea0557bb1f4ef0eefc8389cf780c7dcdaa5038399345748d3f6"} Oct 31 13:26:21.008055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:21.007572 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" event={"ID":"a7abb12e-61a4-42bc-ae37-e6120396e413","Type":"ContainerStarted","Data":"393948bd51fba651abadb1d9aa44896431990aaa2d93c3c67e1bd85f53fcae5e"} Oct 31 13:26:21.033210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:21.033160 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" podStartSLOduration=2.033145416 podStartE2EDuration="2.033145416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:21.02974904 +0000 UTC m=+3275.207467027" watchObservedRunningTime="2026-10-31 13:26:21.033145416 +0000 UTC m=+3275.210863435" Oct 31 13:26:22.020692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:22.020654 5074 generic.go:356] "Generic (PLEG): container finished" podID="a7abb12e-61a4-42bc-ae37-e6120396e413" containerID="d8751971e63afea0557bb1f4ef0eefc8389cf780c7dcdaa5038399345748d3f6" exitCode=0 Oct 31 13:26:22.021099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:22.020731 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" event={"ID":"a7abb12e-61a4-42bc-ae37-e6120396e413","Type":"ContainerDied","Data":"d8751971e63afea0557bb1f4ef0eefc8389cf780c7dcdaa5038399345748d3f6"} Oct 31 13:26:23.227837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.227805 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:23.286713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.280237 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-xks-ca\") pod \"a7abb12e-61a4-42bc-ae37-e6120396e413\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " Oct 31 13:26:23.286713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.280567 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7abb12e-61a4-42bc-ae37-e6120396e413-kube-api-access-6lx75\" (UniqueName: \"kubernetes.io/projected/a7abb12e-61a4-42bc-ae37-e6120396e413-kube-api-access-6lx75\") pod \"a7abb12e-61a4-42bc-ae37-e6120396e413\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " Oct 31 13:26:23.372712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.366095 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-openshift-ca\") pod \"a7abb12e-61a4-42bc-ae37-e6120396e413\" (UID: \"a7abb12e-61a4-42bc-ae37-e6120396e413\") " Oct 31 13:26:23.373349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.373311 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-openshift-ca" pod "a7abb12e-61a4-42bc-ae37-e6120396e413" (UID: "a7abb12e-61a4-42bc-ae37-e6120396e413"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:23.376102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.376053 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-xks-ca" pod "a7abb12e-61a4-42bc-ae37-e6120396e413" (UID: "a7abb12e-61a4-42bc-ae37-e6120396e413"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:26:23.376335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.376305 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:23.376427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.376338 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7abb12e-61a4-42bc-ae37-e6120396e413-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:23.396550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.394387 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7abb12e-61a4-42bc-ae37-e6120396e413-kube-api-access-6lx75" pod "a7abb12e-61a4-42bc-ae37-e6120396e413" (UID: "a7abb12e-61a4-42bc-ae37-e6120396e413"). InnerVolumeSpecName "kube-api-access-6lx75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:23.478298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:23.478258 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lx75\" (UniqueName: \"kubernetes.io/projected/a7abb12e-61a4-42bc-ae37-e6120396e413-kube-api-access-6lx75\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:24.038918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:24.038886 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" Oct 31 13:26:24.039072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:24.038918 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5" event={"ID":"a7abb12e-61a4-42bc-ae37-e6120396e413","Type":"ContainerDied","Data":"393948bd51fba651abadb1d9aa44896431990aaa2d93c3c67e1bd85f53fcae5e"} Oct 31 13:26:24.039072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:24.038966 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="393948bd51fba651abadb1d9aa44896431990aaa2d93c3c67e1bd85f53fcae5e" Oct 31 13:26:25.128563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:25.128525 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv"] Oct 31 13:26:25.133541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:25.133508 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d63f4f10-2kknv"] Oct 31 13:26:25.941472 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:25.941433 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5b81f6-c8b7-4435-8192-8260c0f10a45" path="/var/lib/kubelet/pods/4f5b81f6-c8b7-4435-8192-8260c0f10a45/volumes" Oct 31 13:26:25.988327 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:26:25.988288 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:26:26.954555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:26.954512 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk"] Oct 31 13:26:26.955530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:26.955503 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a7abb12e-61a4-42bc-ae37-e6120396e413" containerName="revoke-keys" Oct 31 13:26:26.955530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:26.955531 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7abb12e-61a4-42bc-ae37-e6120396e413" containerName="revoke-keys" Oct 31 13:26:26.955825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:26.955808 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7abb12e-61a4-42bc-ae37-e6120396e413" containerName="revoke-keys" Oct 31 13:26:26.960082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:26.960061 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:26.968377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:26.967786 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4023ad-openshift-default-dockercfg-n4px5\"" Oct 31 13:26:26.979772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:26.979742 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk"] Oct 31 13:26:27.054055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054012 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12278131-c5a5-4c02-8544-7a943a5c93bd-istiod-ca-cert\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054135 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-data\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054207 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmtt\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-kube-api-access-pkmtt\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054301 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-credential-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054345 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054491 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-token\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054622 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-podinfo\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054733 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-certs\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.054817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.054800 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-envoy\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.164905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.164855 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-certs\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.174662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.165688 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-envoy\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.175597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.175561 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12278131-c5a5-4c02-8544-7a943a5c93bd-istiod-ca-cert\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.176647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.176606 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-data\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.176839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.176819 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmtt\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-kube-api-access-pkmtt\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.177032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.177017 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-credential-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.180762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.180735 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.180992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.180969 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-credential-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.181293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.165563 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-certs\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.181430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.181039 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-token\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.187591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.181569 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-socket\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.187831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.180185 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-data\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.187939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.176510 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12278131-c5a5-4c02-8544-7a943a5c93bd-istiod-ca-cert\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.188027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.181664 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-podinfo\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.218666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.216109 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmtt\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-kube-api-access-pkmtt\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.246477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.246434 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-envoy\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.262337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.262301 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-podinfo\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.273369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.262971 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-token\") pod \"e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.281837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.281808 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:27.575834 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:26:27.575797 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12278131_c5a5_4c02_8544_7a943a5c93bd.slice/crio-c0b076dcabb780729e726facdd445c161f24dc6620a2b43bb5e2d9ed3795e857 WatchSource:0}: Error finding container c0b076dcabb780729e726facdd445c161f24dc6620a2b43bb5e2d9ed3795e857: Status 404 returned error can't find the container with id c0b076dcabb780729e726facdd445c161f24dc6620a2b43bb5e2d9ed3795e857 Oct 31 13:26:27.576064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.576042 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk"] Oct 31 13:26:27.580100 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.580058 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:27.580213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.580155 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:27.580213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:27.580199 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:28.080326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:28.080284 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" event={"ID":"12278131-c5a5-4c02-8544-7a943a5c93bd","Type":"ContainerStarted","Data":"690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277"} Oct 31 13:26:28.081372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:28.081339 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" event={"ID":"12278131-c5a5-4c02-8544-7a943a5c93bd","Type":"ContainerStarted","Data":"c0b076dcabb780729e726facdd445c161f24dc6620a2b43bb5e2d9ed3795e857"} Oct 31 13:26:28.110654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:28.110573 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podStartSLOduration=2.110551837 podStartE2EDuration="2.110551837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:28.103796111 +0000 UTC m=+3282.281514121" watchObservedRunningTime="2026-10-31 13:26:28.110551837 +0000 UTC m=+3282.288269821" Oct 31 13:26:28.283278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:28.283240 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:28.284525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:28.284496 5074 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 13:26:28.284689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:28.284548 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 13:26:29.283653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:29.283066 5074 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 13:26:29.283653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:29.283131 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 13:26:30.104906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.104849 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z"] Oct 31 13:26:30.110942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.110909 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.115064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.115024 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z"] Oct 31 13:26:30.115217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.115114 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:26:30.116852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.116827 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:26:30.116947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.116890 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:26:30.116947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.116840 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:26:30.161169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.161136 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbmq\" (UniqueName: \"kubernetes.io/projected/0eec3c22-4741-4d2d-b198-471a1574fb9f-kube-api-access-wfbmq\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.161399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.161194 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.161399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.161306 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.366716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.361074 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbmq\" (UniqueName: \"kubernetes.io/projected/0eec3c22-4741-4d2d-b198-471a1574fb9f-kube-api-access-wfbmq\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.366716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.361160 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.366716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.361239 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.366716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.363438 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.406937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.379889 5074 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 13:26:30.406937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.379978 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 13:26:30.420939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.420905 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbmq\" (UniqueName: \"kubernetes.io/projected/0eec3c22-4741-4d2d-b198-471a1574fb9f-kube-api-access-wfbmq\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.427349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.427319 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9fe777c-4kd2z\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.448025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.448000 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:30.805687 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:26:30.805358 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eec3c22_4741_4d2d_b198_471a1574fb9f.slice/crio-875d20f06ac2ea1fb17f0a3c4b197b447b377cc95846e993fb9af4481a0ca26d WatchSource:0}: Error finding container 875d20f06ac2ea1fb17f0a3c4b197b447b377cc95846e993fb9af4481a0ca26d: Status 404 returned error can't find the container with id 875d20f06ac2ea1fb17f0a3c4b197b447b377cc95846e993fb9af4481a0ca26d Oct 31 13:26:30.808504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:30.808470 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z"] Oct 31 13:26:31.217247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:31.217205 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" event={"ID":"0eec3c22-4741-4d2d-b198-471a1574fb9f","Type":"ContainerStarted","Data":"875d20f06ac2ea1fb17f0a3c4b197b447b377cc95846e993fb9af4481a0ca26d"} Oct 31 13:26:31.298658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:31.283102 5074 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 13:26:31.298658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:31.283181 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 13:26:32.230479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:32.230441 5074 generic.go:356] "Generic (PLEG): container finished" podID="0eec3c22-4741-4d2d-b198-471a1574fb9f" containerID="0e7f2370786c50f1085e61a282f4693432eeb0bb73ab05cf8ef30ffdd9748fcc" exitCode=0 Oct 31 13:26:32.231000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:32.230531 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" event={"ID":"0eec3c22-4741-4d2d-b198-471a1574fb9f","Type":"ContainerDied","Data":"0e7f2370786c50f1085e61a282f4693432eeb0bb73ab05cf8ef30ffdd9748fcc"} Oct 31 13:26:32.284023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:32.283985 5074 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 13:26:32.284308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:32.284276 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 13:26:33.297702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.297018 5074 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 13:26:33.297702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.297080 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 13:26:33.644123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.644091 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:33.801250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.801078 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eec3c22-4741-4d2d-b198-471a1574fb9f-kube-api-access-wfbmq\" (UniqueName: \"kubernetes.io/projected/0eec3c22-4741-4d2d-b198-471a1574fb9f-kube-api-access-wfbmq\") pod \"0eec3c22-4741-4d2d-b198-471a1574fb9f\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " Oct 31 13:26:33.801250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.801203 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-xks-ca\") pod \"0eec3c22-4741-4d2d-b198-471a1574fb9f\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " Oct 31 13:26:33.801536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.801268 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-openshift-ca\") pod \"0eec3c22-4741-4d2d-b198-471a1574fb9f\" (UID: \"0eec3c22-4741-4d2d-b198-471a1574fb9f\") " Oct 31 13:26:33.802106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.802076 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-openshift-ca" pod "0eec3c22-4741-4d2d-b198-471a1574fb9f" (UID: "0eec3c22-4741-4d2d-b198-471a1574fb9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:33.804364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.804336 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-xks-ca" pod "0eec3c22-4741-4d2d-b198-471a1574fb9f" (UID: "0eec3c22-4741-4d2d-b198-471a1574fb9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:26:33.804716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.804694 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eec3c22-4741-4d2d-b198-471a1574fb9f-kube-api-access-wfbmq" pod "0eec3c22-4741-4d2d-b198-471a1574fb9f" (UID: "0eec3c22-4741-4d2d-b198-471a1574fb9f"). InnerVolumeSpecName "kube-api-access-wfbmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:33.903370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.903275 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbmq\" (UniqueName: \"kubernetes.io/projected/0eec3c22-4741-4d2d-b198-471a1574fb9f-kube-api-access-wfbmq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:33.903370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.903313 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:33.903370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:33.903327 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec3c22-4741-4d2d-b198-471a1574fb9f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:34.253578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:34.253542 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" Oct 31 13:26:34.253889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:34.253565 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z" event={"ID":"0eec3c22-4741-4d2d-b198-471a1574fb9f","Type":"ContainerDied","Data":"875d20f06ac2ea1fb17f0a3c4b197b447b377cc95846e993fb9af4481a0ca26d"} Oct 31 13:26:34.254072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:34.254053 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="875d20f06ac2ea1fb17f0a3c4b197b447b377cc95846e993fb9af4481a0ca26d" Oct 31 13:26:34.282760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:34.282730 5074 patch_prober.go:29] interesting pod/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 13:26:34.282995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:34.282968 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 13:26:35.288314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:35.288282 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:35.288885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:35.288766 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:35.289487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:35.289464 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:26:36.042483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:36.042442 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn"] Oct 31 13:26:36.049779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:36.049754 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df97353d-rxcwn"] Oct 31 13:26:37.941883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:37.941845 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2225a273-83c1-44e7-ad99-abb91a537c00" path="/var/lib/kubelet/pods/2225a273-83c1-44e7-ad99-abb91a537c00/volumes" Oct 31 13:26:40.321377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:40.321338 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm"] Oct 31 13:26:40.321827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:40.321551 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" containerID="cri-o://3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa" gracePeriod=30 Oct 31 13:26:44.867684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:44.867620 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:26:44.930793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:44.930747 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:26:45.625898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.625866 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 13:26:45.635646 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635598 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-certs\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.635814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635678 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-data\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.635814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635702 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-kube-api-access-kc8zh\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-kube-api-access-kc8zh\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.635814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635732 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-socket\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.635814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635767 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73604a4f-c54f-4944-a4b1-6e7519350578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73604a4f-c54f-4944-a4b1-6e7519350578-istio-podinfo\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.636020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635830 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73604a4f-c54f-4944-a4b1-6e7519350578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73604a4f-c54f-4944-a4b1-6e7519350578-istiod-ca-cert\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.636020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635892 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-certs" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:26:45.636020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.635905 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-istio-token\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-istio-token\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.636176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.636026 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-socket" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:26:45.636176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.636138 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-envoy\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.636176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.636031 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-data" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:26:45.636326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.636265 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-credential-socket\") pod \"73604a4f-c54f-4944-a4b1-6e7519350578\" (UID: \"73604a4f-c54f-4944-a4b1-6e7519350578\") " Oct 31 13:26:45.636378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.636356 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73604a4f-c54f-4944-a4b1-6e7519350578-istiod-ca-cert" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:45.636563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.636535 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-credential-socket" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:26:45.637145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.637120 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.637233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.637150 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.637233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.637164 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.637233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.637181 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.637233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.637196 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73604a4f-c54f-4944-a4b1-6e7519350578-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.638397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.638366 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73604a4f-c54f-4944-a4b1-6e7519350578-istio-podinfo" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:26:45.638768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.638748 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-kube-api-access-kc8zh" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "kube-api-access-kc8zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:45.638838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.638778 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-istio-token" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:45.638967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.638952 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-envoy" pod "73604a4f-c54f-4944-a4b1-6e7519350578" (UID: "73604a4f-c54f-4944-a4b1-6e7519350578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:26:45.738647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.738610 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.738819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.738657 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73604a4f-c54f-4944-a4b1-6e7519350578-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.738819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.738669 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8zh\" (UniqueName: \"kubernetes.io/projected/73604a4f-c54f-4944-a4b1-6e7519350578-kube-api-access-kc8zh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.738819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.738681 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73604a4f-c54f-4944-a4b1-6e7519350578-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:45.954768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.954675 5074 generic.go:356] "Generic (PLEG): container finished" podID="73604a4f-c54f-4944-a4b1-6e7519350578" containerID="3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa" exitCode=0 Oct 31 13:26:45.954768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.954745 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" Oct 31 13:26:45.955278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.954760 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" event={"ID":"73604a4f-c54f-4944-a4b1-6e7519350578","Type":"ContainerDied","Data":"3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa"} Oct 31 13:26:45.955278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.954810 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm" event={"ID":"73604a4f-c54f-4944-a4b1-6e7519350578","Type":"ContainerDied","Data":"46ee1d5ff55c69b626a60bd865df446cc34f3dc48ce24889ffa8e41f9927bea5"} Oct 31 13:26:45.955278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.954836 5074 scope.go:122] "RemoveContainer" containerID="3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa" Oct 31 13:26:45.967776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.967750 5074 scope.go:122] "RemoveContainer" containerID="3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa" Oct 31 13:26:45.968093 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:26:45.968060 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa\": container with ID starting with 3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa not found: ID does not exist" containerID="3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa" Oct 31 13:26:45.968159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.968099 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa"} err="failed to get container status \"3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa\": rpc error: code = NotFound desc = could not find container \"3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa\": container with ID starting with 3cb8fd3cd42b3ba1eb99e30676f24c48ed0566eb0b261f6b356019b38e5933fa not found: ID does not exist" Oct 31 13:26:45.990604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.990577 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm"] Oct 31 13:26:45.995332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:45.995304 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-hf6vm"] Oct 31 13:26:46.725689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:46.725660 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:26:46.727301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:46.727277 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:26:47.940435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:47.940395 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" path="/var/lib/kubelet/pods/73604a4f-c54f-4944-a4b1-6e7519350578/volumes" Oct 31 13:26:54.595723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:54.595683 5074 scope.go:122] "RemoveContainer" containerID="c444ee929bd025119f976f5f452b56388f64bab934518cdd8881ea8345cff9ad" Oct 31 13:26:54.610744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:26:54.610708 5074 scope.go:122] "RemoveContainer" containerID="a4b0d72db7788417563fd72d43685370f0db06483dd5bd1d0547e6797f37a965" Oct 31 13:26:56.082940 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:26:56.082898 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:27:08.951091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.951054 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz"] Oct 31 13:27:08.954988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.954938 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0eec3c22-4741-4d2d-b198-471a1574fb9f" containerName="revoke-keys" Oct 31 13:27:08.955287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.955265 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eec3c22-4741-4d2d-b198-471a1574fb9f" containerName="revoke-keys" Oct 31 13:27:08.955521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.955444 5074 cpu_manager.go:414] "containerMap: removing container" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" Oct 31 13:27:08.955669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.955623 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" Oct 31 13:27:08.956035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.956018 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="73604a4f-c54f-4944-a4b1-6e7519350578" containerName="istio-proxy" Oct 31 13:27:08.956154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.956143 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eec3c22-4741-4d2d-b198-471a1574fb9f" containerName="revoke-keys" Oct 31 13:27:08.965686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:08.965659 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.010869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.010726 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz"] Oct 31 13:27:09.071005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.070956 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071044 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071130 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071291 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9083d229-6ab4-4749-bfa7-d69cab443a32-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071358 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071530 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071583 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071659 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzgf\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-kube-api-access-7pzgf\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.071786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.071718 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.174016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.173961 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.174225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.174132 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9083d229-6ab4-4749-bfa7-d69cab443a32-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.174311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.174232 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.174734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.174702 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.175134 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.174952 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9083d229-6ab4-4749-bfa7-d69cab443a32-istiod-ca-cert\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.175407 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzgf\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-kube-api-access-7pzgf\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.175520 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.175416 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-credential-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.175705 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.175770 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.175904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.175882 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-data\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.176182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.176132 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-certs\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.176379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.176355 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-socket\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.177234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.177172 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-podinfo\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.177234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.177178 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-envoy\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.187439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.187410 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-token\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.189074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.189050 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzgf\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-kube-api-access-7pzgf\") pod \"e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.309762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.309725 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:09.611881 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:27:09.611829 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9083d229_6ab4_4749_bfa7_d69cab443a32.slice/crio-a30fb6f848f43ba94315f2e5e2d89e235b2cfbeb9ecc6911d25669b6b6d867b9 WatchSource:0}: Error finding container a30fb6f848f43ba94315f2e5e2d89e235b2cfbeb9ecc6911d25669b6b6d867b9: Status 404 returned error can't find the container with id a30fb6f848f43ba94315f2e5e2d89e235b2cfbeb9ecc6911d25669b6b6d867b9 Oct 31 13:27:09.613933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.613891 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz"] Oct 31 13:27:09.615701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.615611 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:09.615825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.615720 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:09.615825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:09.615763 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:10.599979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:10.599925 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" event={"ID":"9083d229-6ab4-4749-bfa7-d69cab443a32","Type":"ContainerStarted","Data":"2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e"} Oct 31 13:27:10.600424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:10.600009 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" event={"ID":"9083d229-6ab4-4749-bfa7-d69cab443a32","Type":"ContainerStarted","Data":"a30fb6f848f43ba94315f2e5e2d89e235b2cfbeb9ecc6911d25669b6b6d867b9"} Oct 31 13:27:10.630252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:10.630187 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podStartSLOduration=2.630174385 podStartE2EDuration="2.630174385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:10.625430266 +0000 UTC m=+3324.803148273" watchObservedRunningTime="2026-10-31 13:27:10.630174385 +0000 UTC m=+3324.807892370" Oct 31 13:27:11.313658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:11.313361 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:11.313658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:11.313483 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 31 13:27:11.313658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:11.313529 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 31 13:27:12.310256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:12.310218 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 31 13:27:12.310839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:12.310289 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 31 13:27:12.997657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:12.978731 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:27:12.997657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:12.978812 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:13.310562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.310460 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 31 13:27:13.310562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.310523 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 31 13:27:13.710866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.710767 5074 generic.go:356] "Generic (PLEG): container finished" podID="cc8ad859-e258-4a60-b804-ab05271038b5" containerID="ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705" exitCode=0 Oct 31 13:27:13.710866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.710833 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" event={"ID":"cc8ad859-e258-4a60-b804-ab05271038b5","Type":"ContainerDied","Data":"ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705"} Oct 31 13:27:13.711379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.711358 5074 scope.go:122] "RemoveContainer" containerID="ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705" Oct 31 13:27:13.713534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.713501 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:13.713662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.713573 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:13.713662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:13.713608 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:14.310200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:14.310150 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 31 13:27:14.310411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:14.310231 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 31 13:27:14.788659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:14.788468 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" event={"ID":"cc8ad859-e258-4a60-b804-ab05271038b5","Type":"ContainerStarted","Data":"87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf"} Oct 31 13:27:15.039569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.039467 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47"] Oct 31 13:27:15.044350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.044319 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3da27e-skz47"] Oct 31 13:27:15.247506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.247461 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:27:15.247713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.247531 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:27:15.247819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.247796 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:27:15.329762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.329067 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 31 13:27:15.329762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.329148 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 31 13:27:15.948429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:15.948391 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d7d150-175f-464a-86b6-1384cf8aac4b" path="/var/lib/kubelet/pods/24d7d150-175f-464a-86b6-1384cf8aac4b/volumes" Oct 31 13:27:16.247494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:16.247313 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:27:16.247494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:16.247396 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:27:16.316213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:16.316173 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 31 13:27:16.316504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:16.316463 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 31 13:27:17.247253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:17.247213 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:27:17.247727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:17.247266 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:27:17.316863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:17.316831 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:17.317387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:17.317357 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:17.318277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:17.318255 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:27:18.265908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:18.265857 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:27:18.266448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:18.265937 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:27:19.247070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:19.247032 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:27:19.247250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:19.247099 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:27:20.247919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:20.247878 5074 patch_prober.go:29] interesting pod/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:27:20.248428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:20.247941 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:27:21.252325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.252291 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:27:21.252895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.252879 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:27:21.253591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.253567 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:27:21.791432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.791389 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np"] Oct 31 13:27:21.798076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.798045 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:21.814481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.814450 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 13:27:21.818523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.818495 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a39960-serving-cert\"" Oct 31 13:27:21.818830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.818805 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 13:27:21.823032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.821935 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:27:21.823032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.822003 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:27:21.829344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.827898 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np"] Oct 31 13:27:21.886928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.886884 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c28c8e-c760-4e94-9530-a4cc7ea2d908-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:21.887150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.886945 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-maas-api-tls\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:21.887150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.887080 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkdp\" (UniqueName: \"kubernetes.io/projected/77c28c8e-c760-4e94-9530-a4cc7ea2d908-kube-api-access-pkkdp\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:21.887442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.887420 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-metrics-certs\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.012665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:21.997745 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-metrics-certs\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.024607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.022141 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c28c8e-c760-4e94-9530-a4cc7ea2d908-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.024607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.022219 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-maas-api-tls\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.024607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.022279 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkdp\" (UniqueName: \"kubernetes.io/projected/77c28c8e-c760-4e94-9530-a4cc7ea2d908-kube-api-access-pkkdp\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.034944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.034885 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c28c8e-c760-4e94-9530-a4cc7ea2d908-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.078706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.068398 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-metrics-certs\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.078706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.075868 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-maas-api-tls\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.078706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.077062 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkdp\" (UniqueName: \"kubernetes.io/projected/77c28c8e-c760-4e94-9530-a4cc7ea2d908-kube-api-access-pkkdp\") pod \"maas-api-e2e-inf-a-a39960-666c9874dd-zv6np\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.121530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.121479 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:22.497362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.497330 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:27:22.503645 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:22.502944 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np"] Oct 31 13:27:23.063414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:23.063370 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" event={"ID":"77c28c8e-c760-4e94-9530-a4cc7ea2d908","Type":"ContainerStarted","Data":"dadd24ef8b50db5c334d3aacfcf97e063a4f01540b910928fdb07b0c3c998161"} Oct 31 13:27:25.046306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:25.046261 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l"] Oct 31 13:27:25.051784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:25.051754 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c057fdf-dmg7l"] Oct 31 13:27:25.942784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:25.942753 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144dce9f-4826-4397-8e96-84ce54f24c3c" path="/var/lib/kubelet/pods/144dce9f-4826-4397-8e96-84ce54f24c3c/volumes" Oct 31 13:27:25.987489 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:27:25.987446 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:27:26.189652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:26.189540 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" event={"ID":"77c28c8e-c760-4e94-9530-a4cc7ea2d908","Type":"ContainerStarted","Data":"f920d8a7e08e2377184418560edceeec7830cf3b3bb10beddbf60855ef18dbe2"} Oct 31 13:27:26.189652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:26.189610 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:26.222652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:26.221698 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" podStartSLOduration=1.855873654 podStartE2EDuration="5.221678838s" totalImagesPullingTime="3.365805184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:26.212104695 +0000 UTC m=+3340.389822678" watchObservedRunningTime="2026-10-31 13:27:26.221678838 +0000 UTC m=+3340.399396827" Oct 31 13:27:32.246595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:32.246561 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:27:48.857657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:48.844992 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:27:51.138658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.134238 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:51.138658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.134659 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:27:51.199844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.199783 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:51.217792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.217747 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 13:27:51.261902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.261852 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:51.262620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.262579 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 31 13:27:51.285066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.285007 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:51.401413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.401288 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:51.444612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.419812 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:51.478465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:51.477723 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:27:54.657574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:54.657536 5074 scope.go:122] "RemoveContainer" containerID="7d4aa6173f5c2c441ce29e8a20473f1808fec09fe54fe8d21347061db03c5695" Oct 31 13:27:54.670728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:54.670699 5074 scope.go:122] "RemoveContainer" containerID="65e04ce2ef20b9dfab0bf392a3df81337af65dc0e1cba1b2c1bdb1ec371a5b17" Oct 31 13:27:54.681208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:54.681186 5074 scope.go:122] "RemoveContainer" containerID="67e59f368080f1a90159e025e9566980f5d145577e72765779c1a5d90e27bcb2" Oct 31 13:27:54.691082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:54.691061 5074 scope.go:122] "RemoveContainer" containerID="295d70da1c9beb496770bf82ec187231e75c4f4b92695a868c889bf06f76f088" Oct 31 13:27:54.703975 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:54.703952 5074 scope.go:122] "RemoveContainer" containerID="5f7e699d940046b99caed3055f42784fb5e760264496909dcaf1d695d0a09255" Oct 31 13:27:55.145577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.145542 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d"] Oct 31 13:27:55.152202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.152178 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.156781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.156237 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:55.157614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.157588 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:27:55.161303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.161273 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d"] Oct 31 13:27:55.278261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.278221 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.278476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.278387 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhjg\" (UniqueName: \"kubernetes.io/projected/7d23f63d-0090-49a8-a3d0-b2cc9706096a-kube-api-access-wnhjg\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.278580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.278551 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.379945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.379907 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.380130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.380030 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhjg\" (UniqueName: \"kubernetes.io/projected/7d23f63d-0090-49a8-a3d0-b2cc9706096a-kube-api-access-wnhjg\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.380175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.380146 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.380685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.380665 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.382612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.382589 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.393365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.393334 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhjg\" (UniqueName: \"kubernetes.io/projected/7d23f63d-0090-49a8-a3d0-b2cc9706096a-kube-api-access-wnhjg\") pod \"maas-api-revoke-sub-61ae9733-rtn8d\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.477896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.477859 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:55.789106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.789071 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d"] Oct 31 13:27:55.798159 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:27:55.798123 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d23f63d_0090_49a8_a3d0_b2cc9706096a.slice/crio-93b5b6a73f05b8ee79f337d432ec91a6a1fe270b92d6602867bd52ebadb3175a WatchSource:0}: Error finding container 93b5b6a73f05b8ee79f337d432ec91a6a1fe270b92d6602867bd52ebadb3175a: Status 404 returned error can't find the container with id 93b5b6a73f05b8ee79f337d432ec91a6a1fe270b92d6602867bd52ebadb3175a Oct 31 13:27:55.831561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:55.831532 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" event={"ID":"7d23f63d-0090-49a8-a3d0-b2cc9706096a","Type":"ContainerStarted","Data":"93b5b6a73f05b8ee79f337d432ec91a6a1fe270b92d6602867bd52ebadb3175a"} Oct 31 13:27:56.009251 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:27:56.009174 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:27:56.843325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:56.843288 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" event={"ID":"7d23f63d-0090-49a8-a3d0-b2cc9706096a","Type":"ContainerStarted","Data":"eb1c6ab10ca369fd30392e1351efbfbdc425aa5fd758a6a0d0bd253bcc120726"} Oct 31 13:27:56.867270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:56.867220 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" podStartSLOduration=1.867205547 podStartE2EDuration="1.867205547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:56.862522536 +0000 UTC m=+3371.040240521" watchObservedRunningTime="2026-10-31 13:27:56.867205547 +0000 UTC m=+3371.044923531" Oct 31 13:27:57.871088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:57.871050 5074 generic.go:356] "Generic (PLEG): container finished" podID="7d23f63d-0090-49a8-a3d0-b2cc9706096a" containerID="eb1c6ab10ca369fd30392e1351efbfbdc425aa5fd758a6a0d0bd253bcc120726" exitCode=0 Oct 31 13:27:57.871674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:57.871649 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" event={"ID":"7d23f63d-0090-49a8-a3d0-b2cc9706096a","Type":"ContainerDied","Data":"eb1c6ab10ca369fd30392e1351efbfbdc425aa5fd758a6a0d0bd253bcc120726"} Oct 31 13:27:59.079770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.079738 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:59.157258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.157164 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-xks-ca\") pod \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " Oct 31 13:27:59.157416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.157363 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d23f63d-0090-49a8-a3d0-b2cc9706096a-kube-api-access-wnhjg\" (UniqueName: \"kubernetes.io/projected/7d23f63d-0090-49a8-a3d0-b2cc9706096a-kube-api-access-wnhjg\") pod \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " Oct 31 13:27:59.157615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.157591 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-openshift-ca\") pod \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\" (UID: \"7d23f63d-0090-49a8-a3d0-b2cc9706096a\") " Oct 31 13:27:59.158066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.158037 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-openshift-ca" pod "7d23f63d-0090-49a8-a3d0-b2cc9706096a" (UID: "7d23f63d-0090-49a8-a3d0-b2cc9706096a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:59.158618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.158590 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:59.159753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.159728 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d23f63d-0090-49a8-a3d0-b2cc9706096a-kube-api-access-wnhjg" pod "7d23f63d-0090-49a8-a3d0-b2cc9706096a" (UID: "7d23f63d-0090-49a8-a3d0-b2cc9706096a"). InnerVolumeSpecName "kube-api-access-wnhjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:59.161848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.161820 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-xks-ca" pod "7d23f63d-0090-49a8-a3d0-b2cc9706096a" (UID: "7d23f63d-0090-49a8-a3d0-b2cc9706096a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:59.260128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.260090 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d23f63d-0090-49a8-a3d0-b2cc9706096a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:59.260128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.260129 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhjg\" (UniqueName: \"kubernetes.io/projected/7d23f63d-0090-49a8-a3d0-b2cc9706096a-kube-api-access-wnhjg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:59.892119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.892086 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" Oct 31 13:27:59.892119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.892111 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d" event={"ID":"7d23f63d-0090-49a8-a3d0-b2cc9706096a","Type":"ContainerDied","Data":"93b5b6a73f05b8ee79f337d432ec91a6a1fe270b92d6602867bd52ebadb3175a"} Oct 31 13:27:59.892364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:27:59.892185 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b5b6a73f05b8ee79f337d432ec91a6a1fe270b92d6602867bd52ebadb3175a" Oct 31 13:28:05.833881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.833847 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c"] Oct 31 13:28:05.834699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.834682 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7d23f63d-0090-49a8-a3d0-b2cc9706096a" containerName="revoke-keys" Oct 31 13:28:05.834699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.834699 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d23f63d-0090-49a8-a3d0-b2cc9706096a" containerName="revoke-keys" Oct 31 13:28:05.834854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.834844 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d23f63d-0090-49a8-a3d0-b2cc9706096a" containerName="revoke-keys" Oct 31 13:28:05.839150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.839132 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:05.856747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.856718 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c"] Oct 31 13:28:05.901747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.901709 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:28:05.901930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.901850 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:28:05.926134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.926098 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljkx\" (UniqueName: \"kubernetes.io/projected/236819fa-e461-49bd-aab5-f01761c30ec8-kube-api-access-nljkx\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:05.926291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.926266 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:05.926433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:05.926416 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.027928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.027891 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljkx\" (UniqueName: \"kubernetes.io/projected/236819fa-e461-49bd-aab5-f01761c30ec8-kube-api-access-nljkx\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.028132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.028036 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.028214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.028192 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.028928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.028905 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.030498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.030472 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.041563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.041536 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljkx\" (UniqueName: \"kubernetes.io/projected/236819fa-e461-49bd-aab5-f01761c30ec8-kube-api-access-nljkx\") pod \"maas-api-revoke-sub-ceca5bf8-5kf6c\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.153494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.153407 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:06.457297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:06.457270 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c"] Oct 31 13:28:07.150492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:07.150401 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" event={"ID":"236819fa-e461-49bd-aab5-f01761c30ec8","Type":"ContainerStarted","Data":"fbab7917d330bb125508d5ea27990042a328ae03eba52b20cd942d9ce51169ef"} Oct 31 13:28:07.150492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:07.150460 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" event={"ID":"236819fa-e461-49bd-aab5-f01761c30ec8","Type":"ContainerStarted","Data":"94f425af0a9b81106ed40d26597f42b862317cb40262e329d7663af72aeb61db"} Oct 31 13:28:07.219396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:07.219143 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" podStartSLOduration=2.219120004 podStartE2EDuration="2.219120004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:07.197382656 +0000 UTC m=+3381.375100645" watchObservedRunningTime="2026-10-31 13:28:07.219120004 +0000 UTC m=+3381.396837990" Oct 31 13:28:08.225666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:08.219351 5074 generic.go:356] "Generic (PLEG): container finished" podID="236819fa-e461-49bd-aab5-f01761c30ec8" containerID="fbab7917d330bb125508d5ea27990042a328ae03eba52b20cd942d9ce51169ef" exitCode=0 Oct 31 13:28:08.225666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:08.219757 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" event={"ID":"236819fa-e461-49bd-aab5-f01761c30ec8","Type":"ContainerDied","Data":"fbab7917d330bb125508d5ea27990042a328ae03eba52b20cd942d9ce51169ef"} Oct 31 13:28:09.455133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.455105 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:09.548790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.548700 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-xks-ca\") pod \"236819fa-e461-49bd-aab5-f01761c30ec8\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " Oct 31 13:28:09.548981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.548838 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236819fa-e461-49bd-aab5-f01761c30ec8-kube-api-access-nljkx\" (UniqueName: \"kubernetes.io/projected/236819fa-e461-49bd-aab5-f01761c30ec8-kube-api-access-nljkx\") pod \"236819fa-e461-49bd-aab5-f01761c30ec8\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " Oct 31 13:28:09.549064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.549030 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-openshift-ca\") pod \"236819fa-e461-49bd-aab5-f01761c30ec8\" (UID: \"236819fa-e461-49bd-aab5-f01761c30ec8\") " Oct 31 13:28:09.549375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.549347 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-openshift-ca" pod "236819fa-e461-49bd-aab5-f01761c30ec8" (UID: "236819fa-e461-49bd-aab5-f01761c30ec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:09.549750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.549728 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:09.550983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.550958 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-xks-ca" pod "236819fa-e461-49bd-aab5-f01761c30ec8" (UID: "236819fa-e461-49bd-aab5-f01761c30ec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:09.551470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.551453 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236819fa-e461-49bd-aab5-f01761c30ec8-kube-api-access-nljkx" pod "236819fa-e461-49bd-aab5-f01761c30ec8" (UID: "236819fa-e461-49bd-aab5-f01761c30ec8"). InnerVolumeSpecName "kube-api-access-nljkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:09.651442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.651405 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/236819fa-e461-49bd-aab5-f01761c30ec8-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:09.651442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:09.651437 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljkx\" (UniqueName: \"kubernetes.io/projected/236819fa-e461-49bd-aab5-f01761c30ec8-kube-api-access-nljkx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:10.239538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:10.239492 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" event={"ID":"236819fa-e461-49bd-aab5-f01761c30ec8","Type":"ContainerDied","Data":"94f425af0a9b81106ed40d26597f42b862317cb40262e329d7663af72aeb61db"} Oct 31 13:28:10.239780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:10.239600 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f425af0a9b81106ed40d26597f42b862317cb40262e329d7663af72aeb61db" Oct 31 13:28:10.239780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:10.239503 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c" Oct 31 13:28:18.353655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.343856 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9"] Oct 31 13:28:18.353655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.345274 5074 cpu_manager.go:414] "containerMap: removing container" podUID="236819fa-e461-49bd-aab5-f01761c30ec8" containerName="revoke-keys" Oct 31 13:28:18.353655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.345301 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236819fa-e461-49bd-aab5-f01761c30ec8" containerName="revoke-keys" Oct 31 13:28:18.353655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.345500 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="236819fa-e461-49bd-aab5-f01761c30ec8" containerName="revoke-keys" Oct 31 13:28:18.406655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.399766 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9"] Oct 31 13:28:18.406655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.399892 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.449183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.449144 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:28:18.465738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.450208 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:28:18.596253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.596195 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.614391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.614297 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g75r\" (UniqueName: \"kubernetes.io/projected/540edb2f-2875-4b02-99ca-c9afc1161d6a-kube-api-access-5g75r\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.615299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.615269 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.774065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.773551 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.774065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.773768 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g75r\" (UniqueName: \"kubernetes.io/projected/540edb2f-2875-4b02-99ca-c9afc1161d6a-kube-api-access-5g75r\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.774339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.774106 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.799649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.775132 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.799649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.783164 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.799649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.789433 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g75r\" (UniqueName: \"kubernetes.io/projected/540edb2f-2875-4b02-99ca-c9afc1161d6a-kube-api-access-5g75r\") pod \"maas-api-revoke-sub-9b197f1c-t9cb9\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:18.874587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:18.874492 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:19.173820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:19.173349 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9"] Oct 31 13:28:19.531056 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:19.531016 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" event={"ID":"540edb2f-2875-4b02-99ca-c9afc1161d6a","Type":"ContainerStarted","Data":"09bc6a39b6b1578d40792958009a45a40b470b7c1be9677c460d60c6e1c6baa2"} Oct 31 13:28:20.573108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:20.573017 5074 generic.go:356] "Generic (PLEG): container finished" podID="540edb2f-2875-4b02-99ca-c9afc1161d6a" containerID="3abc7e8f26b38379c2df80c88afdfeba4fdf546e05732b9ac89de57f2a66bcc0" exitCode=0 Oct 31 13:28:20.573681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:20.573656 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" event={"ID":"540edb2f-2875-4b02-99ca-c9afc1161d6a","Type":"ContainerDied","Data":"3abc7e8f26b38379c2df80c88afdfeba4fdf546e05732b9ac89de57f2a66bcc0"} Oct 31 13:28:21.848571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.848541 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:21.951212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.951135 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-openshift-ca\") pod \"540edb2f-2875-4b02-99ca-c9afc1161d6a\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " Oct 31 13:28:21.951350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.951323 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-xks-ca\") pod \"540edb2f-2875-4b02-99ca-c9afc1161d6a\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " Oct 31 13:28:21.951407 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.951380 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540edb2f-2875-4b02-99ca-c9afc1161d6a-kube-api-access-5g75r\" (UniqueName: \"kubernetes.io/projected/540edb2f-2875-4b02-99ca-c9afc1161d6a-kube-api-access-5g75r\") pod \"540edb2f-2875-4b02-99ca-c9afc1161d6a\" (UID: \"540edb2f-2875-4b02-99ca-c9afc1161d6a\") " Oct 31 13:28:21.951566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.951542 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-openshift-ca" pod "540edb2f-2875-4b02-99ca-c9afc1161d6a" (UID: "540edb2f-2875-4b02-99ca-c9afc1161d6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:21.952287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.952265 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:21.953509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.953479 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540edb2f-2875-4b02-99ca-c9afc1161d6a-kube-api-access-5g75r" pod "540edb2f-2875-4b02-99ca-c9afc1161d6a" (UID: "540edb2f-2875-4b02-99ca-c9afc1161d6a"). InnerVolumeSpecName "kube-api-access-5g75r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:21.953509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:21.953498 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-xks-ca" pod "540edb2f-2875-4b02-99ca-c9afc1161d6a" (UID: "540edb2f-2875-4b02-99ca-c9afc1161d6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:22.053787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:22.053751 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540edb2f-2875-4b02-99ca-c9afc1161d6a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:22.053787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:22.053786 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g75r\" (UniqueName: \"kubernetes.io/projected/540edb2f-2875-4b02-99ca-c9afc1161d6a-kube-api-access-5g75r\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:22.621954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:22.621921 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" Oct 31 13:28:22.622173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:22.621952 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9" event={"ID":"540edb2f-2875-4b02-99ca-c9afc1161d6a","Type":"ContainerDied","Data":"09bc6a39b6b1578d40792958009a45a40b470b7c1be9677c460d60c6e1c6baa2"} Oct 31 13:28:22.622173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:22.622023 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09bc6a39b6b1578d40792958009a45a40b470b7c1be9677c460d60c6e1c6baa2" Oct 31 13:28:25.985330 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:28:25.985284 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:28:28.387771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.387733 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl"] Oct 31 13:28:28.388553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.388538 5074 cpu_manager.go:414] "containerMap: removing container" podUID="540edb2f-2875-4b02-99ca-c9afc1161d6a" containerName="revoke-keys" Oct 31 13:28:28.388553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.388554 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540edb2f-2875-4b02-99ca-c9afc1161d6a" containerName="revoke-keys" Oct 31 13:28:28.388773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.388761 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="540edb2f-2875-4b02-99ca-c9afc1161d6a" containerName="revoke-keys" Oct 31 13:28:28.393176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.393148 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.401708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.401683 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:28:28.402052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.402033 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:28:28.403461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.403434 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl"] Oct 31 13:28:28.492119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.492076 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.492335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.492211 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2ns\" (UniqueName: \"kubernetes.io/projected/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-kube-api-access-mm2ns\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.492413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.492395 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.594538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.594491 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.594771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.594557 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2ns\" (UniqueName: \"kubernetes.io/projected/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-kube-api-access-mm2ns\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.594771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.594699 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.595399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.595372 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.597247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.597216 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.607976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.607949 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2ns\" (UniqueName: \"kubernetes.io/projected/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-kube-api-access-mm2ns\") pod \"maas-api-revoke-sub-c6708dd1-zgzcl\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:28.727142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:28.727101 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:29.128288 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:28:29.128245 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6f9f30_18c9_47e8_8fa5_ae4de2cd2305.slice/crio-c16169be41b3088338755dbcde17ee7ec059a8c62b4bff489a2d0084a08d1f30 WatchSource:0}: Error finding container c16169be41b3088338755dbcde17ee7ec059a8c62b4bff489a2d0084a08d1f30: Status 404 returned error can't find the container with id c16169be41b3088338755dbcde17ee7ec059a8c62b4bff489a2d0084a08d1f30 Oct 31 13:28:29.130557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:29.130526 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl"] Oct 31 13:28:29.716039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:29.715998 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" event={"ID":"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305","Type":"ContainerStarted","Data":"bf8efa5430fdf8e5c716e1229e131fdc46d678b8c8acea3d54e5c31361d0fa3d"} Oct 31 13:28:29.716558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:29.716085 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" event={"ID":"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305","Type":"ContainerStarted","Data":"c16169be41b3088338755dbcde17ee7ec059a8c62b4bff489a2d0084a08d1f30"} Oct 31 13:28:30.423982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:30.423927 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" podStartSLOduration=2.423911959 podStartE2EDuration="2.423911959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:29.737695404 +0000 UTC m=+3403.915413389" watchObservedRunningTime="2026-10-31 13:28:30.423911959 +0000 UTC m=+3404.601629978" Oct 31 13:28:30.424344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:30.424328 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2"] Oct 31 13:28:30.727691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:30.727654 5074 generic.go:356] "Generic (PLEG): container finished" podID="8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" containerID="bf8efa5430fdf8e5c716e1229e131fdc46d678b8c8acea3d54e5c31361d0fa3d" exitCode=0 Oct 31 13:28:30.728180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:30.727740 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" event={"ID":"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305","Type":"ContainerDied","Data":"bf8efa5430fdf8e5c716e1229e131fdc46d678b8c8acea3d54e5c31361d0fa3d"} Oct 31 13:28:32.035767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.035737 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db"] Oct 31 13:28:32.041439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.041414 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn"] Oct 31 13:28:32.046500 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.046478 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9be9cb68-vt5db"] Oct 31 13:28:32.050927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.050905 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49221240-jsdcn"] Oct 31 13:28:32.053712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.053694 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:32.124334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.124294 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-kube-api-access-mm2ns\" (UniqueName: \"kubernetes.io/projected/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-kube-api-access-mm2ns\") pod \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " Oct 31 13:28:32.124527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.124448 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-xks-ca\") pod \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " Oct 31 13:28:32.124592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.124560 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-openshift-ca\") pod \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\" (UID: \"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305\") " Oct 31 13:28:32.125343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.125310 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-openshift-ca" pod "8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" (UID: "8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:32.127115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.127088 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-xks-ca" pod "8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" (UID: "8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:32.127268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.127241 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-kube-api-access-mm2ns" pod "8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" (UID: "8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305"). InnerVolumeSpecName "kube-api-access-mm2ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:32.226694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.226622 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:32.226694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.226691 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm2ns\" (UniqueName: \"kubernetes.io/projected/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-kube-api-access-mm2ns\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:32.226694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.226703 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:32.749206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.749168 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" Oct 31 13:28:32.749409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.749205 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl" event={"ID":"8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305","Type":"ContainerDied","Data":"c16169be41b3088338755dbcde17ee7ec059a8c62b4bff489a2d0084a08d1f30"} Oct 31 13:28:32.749409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:32.749252 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c16169be41b3088338755dbcde17ee7ec059a8c62b4bff489a2d0084a08d1f30" Oct 31 13:28:33.940919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:33.940881 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24956f45-acac-405b-b473-5850ab98e464" path="/var/lib/kubelet/pods/24956f45-acac-405b-b473-5850ab98e464/volumes" Oct 31 13:28:33.941448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:33.941423 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f27cd2-47ab-4259-91c3-83a951f9d96f" path="/var/lib/kubelet/pods/28f27cd2-47ab-4259-91c3-83a951f9d96f/volumes" Oct 31 13:28:35.425519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.425465 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" podUID="1530f044-02c1-47f5-8daf-43e622687a0c" containerName="payload-processing" containerID="cri-o://8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329" gracePeriod=25 Oct 31 13:28:35.716371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.716344 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:28:35.757170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.757131 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt"] Oct 31 13:28:35.758374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.758342 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" containerName="revoke-keys" Oct 31 13:28:35.758374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.758371 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" containerName="revoke-keys" Oct 31 13:28:35.758576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.758463 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1530f044-02c1-47f5-8daf-43e622687a0c" containerName="payload-processing" Oct 31 13:28:35.758576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.758472 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1530f044-02c1-47f5-8daf-43e622687a0c" containerName="payload-processing" Oct 31 13:28:35.758729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.758684 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1530f044-02c1-47f5-8daf-43e622687a0c" containerName="payload-processing" Oct 31 13:28:35.758729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.758709 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" containerName="revoke-keys" Oct 31 13:28:35.774291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.774152 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt"] Oct 31 13:28:35.774477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.774256 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:35.780734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.780705 5074 generic.go:356] "Generic (PLEG): container finished" podID="1530f044-02c1-47f5-8daf-43e622687a0c" containerID="8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329" exitCode=0 Oct 31 13:28:35.780966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.780763 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" Oct 31 13:28:35.781095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.781067 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" event={"ID":"1530f044-02c1-47f5-8daf-43e622687a0c","Type":"ContainerDied","Data":"8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329"} Oct 31 13:28:35.781231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.781130 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2" event={"ID":"1530f044-02c1-47f5-8daf-43e622687a0c","Type":"ContainerDied","Data":"e3fa8339cf94dd8fa44a7d342e06a49c570c2558df40c5dd1f5f181e8fb4ed7b"} Oct 31 13:28:35.781231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.781169 5074 scope.go:122] "RemoveContainer" containerID="8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329" Oct 31 13:28:35.800451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.800418 5074 scope.go:122] "RemoveContainer" containerID="8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329" Oct 31 13:28:35.800850 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:28:35.800820 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329\": container with ID starting with 8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329 not found: ID does not exist" containerID="8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329" Oct 31 13:28:35.800946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.800861 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329"} err="failed to get container status \"8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329\": rpc error: code = NotFound desc = could not find container \"8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329\": container with ID starting with 8a8294948a740cf5e80291783ea3da5b0fa20f7d5630b620e78287fe3c6f8329 not found: ID does not exist" Oct 31 13:28:35.842887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.842856 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-plugins-config-volume\") pod \"1530f044-02c1-47f5-8daf-43e622687a0c\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " Oct 31 13:28:35.843024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.842981 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-kube-api-access-cllg5\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-kube-api-access-cllg5\") pod \"1530f044-02c1-47f5-8daf-43e622687a0c\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " Oct 31 13:28:35.843125 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.843109 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-provider-credentials\") pod \"1530f044-02c1-47f5-8daf-43e622687a0c\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " Oct 31 13:28:35.843222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.843206 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-routing-overlay-volume\") pod \"1530f044-02c1-47f5-8daf-43e622687a0c\" (UID: \"1530f044-02c1-47f5-8daf-43e622687a0c\") " Oct 31 13:28:35.843295 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.843269 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-plugins-config-volume" pod "1530f044-02c1-47f5-8daf-43e622687a0c" (UID: "1530f044-02c1-47f5-8daf-43e622687a0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:35.843608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.843579 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-routing-overlay-volume" pod "1530f044-02c1-47f5-8daf-43e622687a0c" (UID: "1530f044-02c1-47f5-8daf-43e622687a0c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:35.844020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.843999 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:35.844020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.844021 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1530f044-02c1-47f5-8daf-43e622687a0c-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:35.845418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.845395 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-kube-api-access-cllg5" pod "1530f044-02c1-47f5-8daf-43e622687a0c" (UID: "1530f044-02c1-47f5-8daf-43e622687a0c"). InnerVolumeSpecName "kube-api-access-cllg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:35.845499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.845431 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-provider-credentials" pod "1530f044-02c1-47f5-8daf-43e622687a0c" (UID: "1530f044-02c1-47f5-8daf-43e622687a0c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:35.954601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.945511 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:35.954601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.945583 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:35.954601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.945728 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:35.954601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.945843 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsv65\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-kube-api-access-rsv65\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:35.954601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.946038 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:35.954601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:35.946055 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cllg5\" (UniqueName: \"kubernetes.io/projected/1530f044-02c1-47f5-8daf-43e622687a0c-kube-api-access-cllg5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:36.047914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.047876 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsv65\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-kube-api-access-rsv65\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.048072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.048058 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.048120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.048095 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.048215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.048199 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.049088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.049061 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.049146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.049119 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.050549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.050524 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.060318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.060288 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsv65\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-kube-api-access-rsv65\") pod \"payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.102461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.102432 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.103999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.103968 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2"] Oct 31 13:28:36.108933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.108911 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-842ac9/payload-processing-external-model-e2e-extmod-a-842ac9-c84cchgw2"] Oct 31 13:28:36.399079 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:28:36.399044 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bfbd9_970a_4f2f_85a7_ba5ecc75fb50.slice/crio-2cfff404e3ce9680533b764d358a0b6da7625e4f8dd6e1cb19ad4eab867fc6ad WatchSource:0}: Error finding container 2cfff404e3ce9680533b764d358a0b6da7625e4f8dd6e1cb19ad4eab867fc6ad: Status 404 returned error can't find the container with id 2cfff404e3ce9680533b764d358a0b6da7625e4f8dd6e1cb19ad4eab867fc6ad Oct 31 13:28:36.399549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.399518 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt"] Oct 31 13:28:36.791720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.791671 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" event={"ID":"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50","Type":"ContainerStarted","Data":"4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c"} Oct 31 13:28:36.792251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.791789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" event={"ID":"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50","Type":"ContainerStarted","Data":"2cfff404e3ce9680533b764d358a0b6da7625e4f8dd6e1cb19ad4eab867fc6ad"} Oct 31 13:28:36.792251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.791947 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:36.820912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:36.820861 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" podStartSLOduration=19.820847573 podStartE2EDuration="19.820847573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:36.819839036 +0000 UTC m=+3410.997557021" watchObservedRunningTime="2026-10-31 13:28:36.820847573 +0000 UTC m=+3410.998565556" Oct 31 13:28:37.943263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:37.943225 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1530f044-02c1-47f5-8daf-43e622687a0c" path="/var/lib/kubelet/pods/1530f044-02c1-47f5-8daf-43e622687a0c/volumes" Oct 31 13:28:40.278658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:40.278235 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt"] Oct 31 13:28:40.408726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:40.405699 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2"] Oct 31 13:28:40.932656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:40.932227 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:45.279045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.278999 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" podUID="d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" containerName="payload-processing" containerID="cri-o://4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c" gracePeriod=25 Oct 31 13:28:45.406338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.406289 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" podUID="a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" containerName="payload-processing" containerID="cri-o://2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5" gracePeriod=25 Oct 31 13:28:45.711891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.711854 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:45.809815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.809739 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:28:45.843747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.843711 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx"] Oct 31 13:28:45.844987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.844955 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" containerName="payload-processing" Oct 31 13:28:45.844987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.844983 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" containerName="payload-processing" Oct 31 13:28:45.845210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.845026 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" containerName="payload-processing" Oct 31 13:28:45.845210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.845034 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" containerName="payload-processing" Oct 31 13:28:45.845342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.845269 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" containerName="payload-processing" Oct 31 13:28:45.845342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.845308 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" containerName="payload-processing" Oct 31 13:28:45.850189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.850163 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.850438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.850410 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-provider-credentials\") pod \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " Oct 31 13:28:45.850769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.850734 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-routing-overlay-volume\") pod \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " Oct 31 13:28:45.850875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.850795 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-kube-api-access-rsv65\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-kube-api-access-rsv65\") pod \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " Oct 31 13:28:45.851309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.850957 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-plugins-config-volume\") pod \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\" (UID: \"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50\") " Oct 31 13:28:45.851309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.851211 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-routing-overlay-volume" pod "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" (UID: "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:45.851649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.851607 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-plugins-config-volume" pod "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" (UID: "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:45.851981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.851955 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.852060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.851987 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.855072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.854923 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-kube-api-access-rsv65" pod "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" (UID: "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50"). InnerVolumeSpecName "kube-api-access-rsv65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:45.855602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.855582 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a13836-openshift-default-dockercfg-9t9jk\"" Oct 31 13:28:45.861066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.861042 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx"] Oct 31 13:28:45.873677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.869527 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-provider-credentials" pod "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" (UID: "d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:45.952823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.952789 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-routing-overlay-volume\") pod \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " Oct 31 13:28:45.953031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.952857 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-kube-api-access-r8v9n\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-kube-api-access-r8v9n\") pod \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " Oct 31 13:28:45.953031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.952944 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-provider-credentials\") pod \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " Oct 31 13:28:45.953241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953216 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-plugins-config-volume\") pod \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\" (UID: \"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc\") " Oct 31 13:28:45.953390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953217 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-routing-overlay-volume" pod "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" (UID: "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:45.953477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953457 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-istio-token\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.953558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953489 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c639106-3f06-4fe9-9883-f578170b9015-istiod-ca-cert\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.953558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953540 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-socket\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.953720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953595 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-plugins-config-volume" pod "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" (UID: "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:45.953720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953678 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-credential-socket\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.953843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953808 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c639106-3f06-4fe9-9883-f578170b9015-istio-podinfo\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.953913 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.953893 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-envoy\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.954077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.954055 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-data\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.954206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.954189 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-certs\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.954267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.954253 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4dk\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-kube-api-access-ss4dk\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:45.954390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.954374 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.954448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.954396 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsv65\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-kube-api-access-rsv65\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.954448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.954415 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.954448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.954430 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:45.955299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.955273 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-kube-api-access-r8v9n" pod "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" (UID: "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc"). InnerVolumeSpecName "kube-api-access-r8v9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:45.955766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:45.955746 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-provider-credentials" pod "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" (UID: "a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:46.046274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.046240 5074 generic.go:356] "Generic (PLEG): container finished" podID="a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" containerID="2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5" exitCode=0 Oct 31 13:28:46.046459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.046314 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" Oct 31 13:28:46.046459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.046321 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" event={"ID":"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc","Type":"ContainerDied","Data":"2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5"} Oct 31 13:28:46.046459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.046391 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2" event={"ID":"a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc","Type":"ContainerDied","Data":"286be924aa61e7746d22bf75c32fdd33cc170d4b555fec94bd0c7b4061f6cc2c"} Oct 31 13:28:46.046459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.046422 5074 scope.go:122] "RemoveContainer" containerID="2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5" Oct 31 13:28:46.048251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.048223 5074 generic.go:356] "Generic (PLEG): container finished" podID="d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" containerID="4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c" exitCode=0 Oct 31 13:28:46.048389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.048280 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" Oct 31 13:28:46.048389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.048293 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" event={"ID":"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50","Type":"ContainerDied","Data":"4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c"} Oct 31 13:28:46.048389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.048335 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt" event={"ID":"d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50","Type":"ContainerDied","Data":"2cfff404e3ce9680533b764d358a0b6da7625e4f8dd6e1cb19ad4eab867fc6ad"} Oct 31 13:28:46.055571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.055541 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-certs\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.055789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.055621 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4dk\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-kube-api-access-ss4dk\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.055789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.055722 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-istio-token\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.055789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.055760 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c639106-3f06-4fe9-9883-f578170b9015-istiod-ca-cert\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.055901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.055821 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-socket\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.055901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.055887 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-credential-socket\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.055999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.055964 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c639106-3f06-4fe9-9883-f578170b9015-istio-podinfo\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.056119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.056022 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-envoy\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.056194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.056133 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-data\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.056372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.056350 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8v9n\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-kube-api-access-r8v9n\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:46.056472 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.056378 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:46.057517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.057494 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-certs\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.057753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.057730 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-socket\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.057864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.057702 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-credential-socket\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.058205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.057982 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-data\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.058282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.058229 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c639106-3f06-4fe9-9883-f578170b9015-istiod-ca-cert\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.060477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.060421 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-envoy\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.073398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.072387 5074 scope.go:122] "RemoveContainer" containerID="2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5" Oct 31 13:28:46.073870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.073841 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c639106-3f06-4fe9-9883-f578170b9015-istio-podinfo\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.074004 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:28:46.073928 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5\": container with ID starting with 2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5 not found: ID does not exist" containerID="2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5" Oct 31 13:28:46.074004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.073966 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5"} err="failed to get container status \"2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5\": rpc error: code = NotFound desc = could not find container \"2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5\": container with ID starting with 2d10c0bcc8b4d512e036e6f2f9bd6edd4611b32e37aff2d538e66c40667c02a5 not found: ID does not exist" Oct 31 13:28:46.074004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.073992 5074 scope.go:122] "RemoveContainer" containerID="4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c" Oct 31 13:28:46.076762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.076470 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-istio-token\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.081281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.081244 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4dk\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-kube-api-access-ss4dk\") pod \"e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.089546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.089523 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt"] Oct 31 13:28:46.094124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.094098 5074 scope.go:122] "RemoveContainer" containerID="4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c" Oct 31 13:28:46.094480 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:28:46.094459 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c\": container with ID starting with 4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c not found: ID does not exist" containerID="4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c" Oct 31 13:28:46.094560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.094491 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c"} err="failed to get container status \"4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c\": rpc error: code = NotFound desc = could not find container \"4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c\": container with ID starting with 4a125a5735bb4104bd13a7a98f40a04ebace60c439d61a4cc19a4fd02047127c not found: ID does not exist" Oct 31 13:28:46.096037 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.096010 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-557b5jtxt"] Oct 31 13:28:46.107793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.107766 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2"] Oct 31 13:28:46.113096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.113068 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-15a6a2/payload-processing-external-model-e2e-extmod-b-15a6a2-cb4b8jsh2"] Oct 31 13:28:46.198217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.198173 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a13836-openshift-default-dockercfg-9t9jk\"" Oct 31 13:28:46.206339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.206312 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:46.895786 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:28:46.895757 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c639106_3f06_4fe9_9883_f578170b9015.slice/crio-945869b98129baca74ae632a767859b9174af5938c9c6d73b2c7fd32a4250d42 WatchSource:0}: Error finding container 945869b98129baca74ae632a767859b9174af5938c9c6d73b2c7fd32a4250d42: Status 404 returned error can't find the container with id 945869b98129baca74ae632a767859b9174af5938c9c6d73b2c7fd32a4250d42 Oct 31 13:28:46.897325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.897280 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx"] Oct 31 13:28:46.899943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.899903 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:46.900039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.899992 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:46.900122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:46.900042 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:47.080991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:47.080955 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" event={"ID":"8c639106-3f06-4fe9-9883-f578170b9015","Type":"ContainerStarted","Data":"945869b98129baca74ae632a767859b9174af5938c9c6d73b2c7fd32a4250d42"} Oct 31 13:28:47.972328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:47.972288 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc" path="/var/lib/kubelet/pods/a95d4cfd-3c3f-44d5-bb9f-39536dd3f6dc/volumes" Oct 31 13:28:47.977079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:47.977042 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50" path="/var/lib/kubelet/pods/d02bfbd9-970a-4f2f-85a7-ba5ecc75fb50/volumes" Oct 31 13:28:48.095694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:48.095607 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" event={"ID":"8c639106-3f06-4fe9-9883-f578170b9015","Type":"ContainerStarted","Data":"9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e"} Oct 31 13:28:48.125210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:48.125156 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podStartSLOduration=45.125140669 podStartE2EDuration="45.125140669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:48.120939477 +0000 UTC m=+3422.298657473" watchObservedRunningTime="2026-10-31 13:28:48.125140669 +0000 UTC m=+3422.302858654" Oct 31 13:28:48.206589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:48.206490 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:48.212732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:48.212702 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:49.206658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:49.189781 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:49.230661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:49.228134 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:28:51.843234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:51.843191 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm"] Oct 31 13:28:51.848584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:51.848554 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:51.857461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:51.857433 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm"] Oct 31 13:28:51.857880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:51.857815 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a13836-dockercfg-4j9gf\"" Oct 31 13:28:51.858314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:51.858288 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a13836\"" Oct 31 13:28:51.878464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:51.878425 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fcd3d7-75a1-4486-bb38-e102922aaada-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:51.878647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:51.878613 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtpp\" (UniqueName: \"kubernetes.io/projected/54fcd3d7-75a1-4486-bb38-e102922aaada-kube-api-access-cmtpp\") pod \"payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:52.040657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:52.011443 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtpp\" (UniqueName: \"kubernetes.io/projected/54fcd3d7-75a1-4486-bb38-e102922aaada-kube-api-access-cmtpp\") pod \"payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:52.041617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:52.041568 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fcd3d7-75a1-4486-bb38-e102922aaada-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:52.051330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:52.051299 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fcd3d7-75a1-4486-bb38-e102922aaada-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:52.165705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:52.157667 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtpp\" (UniqueName: \"kubernetes.io/projected/54fcd3d7-75a1-4486-bb38-e102922aaada-kube-api-access-cmtpp\") pod \"payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:52.199730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:52.199687 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:52.692614 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:28:52.692579 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fcd3d7_75a1_4486_bb38_e102922aaada.slice/crio-1c8562e07049f50da8389aa422549544641c86f90bf55425470a243f8647ba74 WatchSource:0}: Error finding container 1c8562e07049f50da8389aa422549544641c86f90bf55425470a243f8647ba74: Status 404 returned error can't find the container with id 1c8562e07049f50da8389aa422549544641c86f90bf55425470a243f8647ba74 Oct 31 13:28:52.694816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:52.694778 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm"] Oct 31 13:28:53.332865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:53.332825 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" event={"ID":"54fcd3d7-75a1-4486-bb38-e102922aaada","Type":"ContainerStarted","Data":"2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975"} Oct 31 13:28:53.333346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:53.332915 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:28:53.333346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:53.332933 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" event={"ID":"54fcd3d7-75a1-4486-bb38-e102922aaada","Type":"ContainerStarted","Data":"1c8562e07049f50da8389aa422549544641c86f90bf55425470a243f8647ba74"} Oct 31 13:28:53.368323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:53.368256 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" podStartSLOduration=2.36823684 podStartE2EDuration="2.36823684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:53.362799934 +0000 UTC m=+3427.540517932" watchObservedRunningTime="2026-10-31 13:28:53.36823684 +0000 UTC m=+3427.545954825" Oct 31 13:28:54.762365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:54.762328 5074 scope.go:122] "RemoveContainer" containerID="e960757077826e6ea651776b9cd91534add634df39c963752fff3d5419976c5a" Oct 31 13:28:54.773741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:54.773719 5074 scope.go:122] "RemoveContainer" containerID="0452e5d97756284d6bd488c8cb156faa8631358f129338699e68dee39c236337" Oct 31 13:28:56.097239 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:28:56.097161 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:28:59.698956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:28:59.698911 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:29:04.345523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:04.345489 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:29:15.085553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:15.085517 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm"] Oct 31 13:29:15.086031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:15.085749 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" containerID="cri-o://76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c" gracePeriod=30 Oct 31 13:29:20.408539 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.408506 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:29:20.447418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.447384 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-socket\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.447620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.447509 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-kube-api-access-r75gr\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-kube-api-access-r75gr\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.447746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.447671 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-data\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.447746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.447706 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-certs\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.447746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.447711 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-socket" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:20.447918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.447860 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-envoy\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.447997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.447981 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-istio-token\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-istio-token\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.448099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.448082 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b2529d-286c-46e2-adf9-81367705d49a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b2529d-286c-46e2-adf9-81367705d49a-istiod-ca-cert\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.448099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.448084 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-data" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:20.448569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.448544 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-certs" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:20.448794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.448766 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-credential-socket\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.449006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.448583 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b2529d-286c-46e2-adf9-81367705d49a-istiod-ca-cert" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:20.449006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.448923 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-credential-socket" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:20.449176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.449005 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75b2529d-286c-46e2-adf9-81367705d49a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b2529d-286c-46e2-adf9-81367705d49a-istio-podinfo\") pod \"75b2529d-286c-46e2-adf9-81367705d49a\" (UID: \"75b2529d-286c-46e2-adf9-81367705d49a\") " Oct 31 13:29:20.450740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.450711 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-envoy" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:20.451183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451156 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-istio-token" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:20.451401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451322 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.451401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451346 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b2529d-286c-46e2-adf9-81367705d49a-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.451401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451361 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.451401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451375 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.451401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451388 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.451600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451401 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b2529d-286c-46e2-adf9-81367705d49a-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.451661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451627 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b2529d-286c-46e2-adf9-81367705d49a-istio-podinfo" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:20.451734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.451711 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl"] Oct 31 13:29:20.452589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.452562 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-kube-api-access-r75gr" pod "75b2529d-286c-46e2-adf9-81367705d49a" (UID: "75b2529d-286c-46e2-adf9-81367705d49a"). InnerVolumeSpecName "kube-api-access-r75gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:20.453754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.453731 5074 cpu_manager.go:414] "containerMap: removing container" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" Oct 31 13:29:20.453973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.453756 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" Oct 31 13:29:20.454059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.454047 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b2529d-286c-46e2-adf9-81367705d49a" containerName="istio-proxy" Oct 31 13:29:20.458958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.458937 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.469209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.465542 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a13836-serving-cert\"" Oct 31 13:29:20.470930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.470901 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl"] Oct 31 13:29:20.552628 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.552594 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015076d4-f1f7-47af-94fb-09395734c846-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.553012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.552989 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-maas-api-tls\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.553116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.553038 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwk8n\" (UniqueName: \"kubernetes.io/projected/015076d4-f1f7-47af-94fb-09395734c846-kube-api-access-qwk8n\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.553195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.553178 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-metrics-certs\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.553341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.553324 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.553380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.553345 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b2529d-286c-46e2-adf9-81367705d49a-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.553380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.553354 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r75gr\" (UniqueName: \"kubernetes.io/projected/75b2529d-286c-46e2-adf9-81367705d49a-kube-api-access-r75gr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:20.657730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.657575 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-maas-api-tls\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.657730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.657669 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwk8n\" (UniqueName: \"kubernetes.io/projected/015076d4-f1f7-47af-94fb-09395734c846-kube-api-access-qwk8n\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.677894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.657762 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-metrics-certs\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.677894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.657838 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015076d4-f1f7-47af-94fb-09395734c846-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.677894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.658789 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015076d4-f1f7-47af-94fb-09395734c846-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.677894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.667305 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-metrics-certs\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.697653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.680497 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-maas-api-tls\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.697653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.687110 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwk8n\" (UniqueName: \"kubernetes.io/projected/015076d4-f1f7-47af-94fb-09395734c846-kube-api-access-qwk8n\") pod \"maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.783480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.783389 5074 generic.go:356] "Generic (PLEG): container finished" podID="75b2529d-286c-46e2-adf9-81367705d49a" containerID="76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c" exitCode=0 Oct 31 13:29:20.783480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.783459 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" Oct 31 13:29:20.783859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.783477 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" event={"ID":"75b2529d-286c-46e2-adf9-81367705d49a","Type":"ContainerDied","Data":"76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c"} Oct 31 13:29:20.783859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.783608 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm" event={"ID":"75b2529d-286c-46e2-adf9-81367705d49a","Type":"ContainerDied","Data":"fc9cf4bc8911b18b3436cad41cc81b4692ce12415dbecdbccc9cf1a0c2146934"} Oct 31 13:29:20.783859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.783674 5074 scope.go:122] "RemoveContainer" containerID="76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c" Oct 31 13:29:20.800627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.800602 5074 scope.go:122] "RemoveContainer" containerID="76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c" Oct 31 13:29:20.800946 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:29:20.800925 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c\": container with ID starting with 76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c not found: ID does not exist" containerID="76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c" Oct 31 13:29:20.801042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.800954 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c"} err="failed to get container status \"76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c\": rpc error: code = NotFound desc = could not find container \"76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c\": container with ID starting with 76de486ffe78e4659304c4b62cbf03cdc7d8ae69a8ea4cd45c6b09df47cd561c not found: ID does not exist" Oct 31 13:29:20.814826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.814724 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:20.819298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.819280 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm"] Oct 31 13:29:20.824458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:20.824438 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-82hwm"] Oct 31 13:29:21.112732 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:29:21.112700 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015076d4_f1f7_47af_94fb_09395734c846.slice/crio-a960f03caa81553a140b0e4a85d4d7a76e72e525d06ecff5121df6e3e996709d WatchSource:0}: Error finding container a960f03caa81553a140b0e4a85d4d7a76e72e525d06ecff5121df6e3e996709d: Status 404 returned error can't find the container with id a960f03caa81553a140b0e4a85d4d7a76e72e525d06ecff5121df6e3e996709d Oct 31 13:29:21.114069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:21.114037 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl"] Oct 31 13:29:21.795613 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:21.795568 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" event={"ID":"015076d4-f1f7-47af-94fb-09395734c846","Type":"ContainerStarted","Data":"a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb"} Oct 31 13:29:21.796566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:21.795658 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:21.796566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:21.795686 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" event={"ID":"015076d4-f1f7-47af-94fb-09395734c846","Type":"ContainerStarted","Data":"a960f03caa81553a140b0e4a85d4d7a76e72e525d06ecff5121df6e3e996709d"} Oct 31 13:29:21.833520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:21.833466 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" podStartSLOduration=4.35917117 podStartE2EDuration="4.833451441s" totalImagesPullingTime="474.280271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:21.829807473 +0000 UTC m=+3456.007525459" watchObservedRunningTime="2026-10-31 13:29:21.833451441 +0000 UTC m=+3456.011169425" Oct 31 13:29:21.945865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:21.945783 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b2529d-286c-46e2-adf9-81367705d49a" path="/var/lib/kubelet/pods/75b2529d-286c-46e2-adf9-81367705d49a/volumes" Oct 31 13:29:25.988798 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:29:25.988752 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:29:27.811693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:27.811622 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:29:51.411454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:51.410546 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk"] Oct 31 13:29:51.411454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:51.410816 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" containerID="cri-o://f4dd03090444c2ad107d1882dd0bd75faae4d22833e096ff10ff68d6ba0a12df" gracePeriod=30 Oct 31 13:29:51.427593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:51.427557 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh"] Oct 31 13:29:51.428194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:51.428148 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" containerID="cri-o://32ba2e750b3936d2830d5b9d43c289df5d2f95dbc39272b55c48564447a01555" gracePeriod=30 Oct 31 13:29:54.683742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.683626 5074 generic.go:356] "Generic (PLEG): container finished" podID="8c639106-3f06-4fe9-9883-f578170b9015" containerID="9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e" exitCode=0 Oct 31 13:29:54.684298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.683880 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" event={"ID":"8c639106-3f06-4fe9-9883-f578170b9015","Type":"ContainerDied","Data":"9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e"} Oct 31 13:29:54.684391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.684370 5074 scope.go:122] "RemoveContainer" containerID="9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e" Oct 31 13:29:54.686912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.686873 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:54.687072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.686967 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:54.687072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.687020 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:54.911327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.911291 5074 scope.go:122] "RemoveContainer" containerID="916f1c8c28cf8354cd96b4b2b43b571d4a5df693f2e0048d292a38d3b5b95938" Oct 31 13:29:54.961666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:54.959948 5074 scope.go:122] "RemoveContainer" containerID="6b15b5e437f2d0a288f99e6d87503ddbd6f0f11619c7d7f787fbfa887a4c80ad" Oct 31 13:29:55.695134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:55.695093 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" event={"ID":"8c639106-3f06-4fe9-9883-f578170b9015","Type":"ContainerStarted","Data":"28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10"} Oct 31 13:29:56.017786 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:29:56.017730 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:29:56.206710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:56.206559 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:29:56.206996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:56.206965 5074 patch_prober.go:29] interesting pod/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 31 13:29:56.207096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:56.207037 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 31 13:29:56.723559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:56.723039 5074 generic.go:356] "Generic (PLEG): container finished" podID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerID="f4dd03090444c2ad107d1882dd0bd75faae4d22833e096ff10ff68d6ba0a12df" exitCode=0 Oct 31 13:29:56.723559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:56.723145 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" event={"ID":"4bae7342-6a60-4145-8420-13c5bdf0beb0","Type":"ContainerDied","Data":"f4dd03090444c2ad107d1882dd0bd75faae4d22833e096ff10ff68d6ba0a12df"} Oct 31 13:29:56.725715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:56.725684 5074 generic.go:356] "Generic (PLEG): container finished" podID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerID="32ba2e750b3936d2830d5b9d43c289df5d2f95dbc39272b55c48564447a01555" exitCode=0 Oct 31 13:29:56.725864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:56.725761 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" event={"ID":"82ba5666-3275-407a-90b6-bcd8adb12e64","Type":"ContainerDied","Data":"32ba2e750b3936d2830d5b9d43c289df5d2f95dbc39272b55c48564447a01555"} Oct 31 13:29:57.024817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.024665 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:29:57.028438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.028419 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:29:57.062946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.062414 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-credential-socket" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.062946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.062513 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-credential-socket\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.062946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.062733 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-socket\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.063251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.062989 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-socket" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.063251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.062756 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-podinfo\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.063251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.063053 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-token\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-token\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.063483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.063465 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bae7342-6a60-4145-8420-13c5bdf0beb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bae7342-6a60-4145-8420-13c5bdf0beb0-istiod-ca-cert\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.063549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.063510 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-envoy\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.066410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.063541 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-data\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.066410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.064360 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-data" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.066410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.064779 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bae7342-6a60-4145-8420-13c5bdf0beb0-istiod-ca-cert" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:57.066410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.065075 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-kube-api-access-ccdhh\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-kube-api-access-ccdhh\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.066410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.065137 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-certs\") pod \"4bae7342-6a60-4145-8420-13c5bdf0beb0\" (UID: \"4bae7342-6a60-4145-8420-13c5bdf0beb0\") " Oct 31 13:29:57.066410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.065488 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-certs" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.066879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.066694 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bae7342-6a60-4145-8420-13c5bdf0beb0-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.066879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.066717 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.066879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.066731 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.066879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.066745 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.066879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.066759 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.067153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.067138 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-envoy" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.067438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.067412 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-token" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:57.068481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.068440 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-podinfo" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.126135 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp"] Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.127433 5074 cpu_manager.go:414] "containerMap: removing container" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.127460 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.127514 5074 cpu_manager.go:414] "containerMap: removing container" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.127524 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.127773 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" containerName="istio-proxy" Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.127806 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" containerName="istio-proxy" Oct 31 13:29:57.134523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.132508 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.137592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.136813 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-kube-api-access-ccdhh" pod "4bae7342-6a60-4145-8420-13c5bdf0beb0" (UID: "4bae7342-6a60-4145-8420-13c5bdf0beb0"). InnerVolumeSpecName "kube-api-access-ccdhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:57.137967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.137940 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-691f1e-serving-cert\"" Oct 31 13:29:57.149263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.149182 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp"] Oct 31 13:29:57.167984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.167952 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-kube-api-access-c5ttt\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-kube-api-access-c5ttt\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168041 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-podinfo\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168094 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ba5666-3275-407a-90b6-bcd8adb12e64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba5666-3275-407a-90b6-bcd8adb12e64-istiod-ca-cert\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168190 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-data\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168231 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-token\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-token\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168270 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-credential-socket\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168344 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-envoy\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168392 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-certs\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168470 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-socket\") pod \"82ba5666-3275-407a-90b6-bcd8adb12e64\" (UID: \"82ba5666-3275-407a-90b6-bcd8adb12e64\") " Oct 31 13:29:57.168499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.168468 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ba5666-3275-407a-90b6-bcd8adb12e64-istiod-ca-cert" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:57.172653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.171145 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-podinfo" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:29:57.172653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.171363 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-data" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.177712 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-credential-socket" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.178943 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-socket" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.179120 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-certs" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187150 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187179 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187194 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187210 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba5666-3275-407a-90b6-bcd8adb12e64-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187226 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bae7342-6a60-4145-8420-13c5bdf0beb0-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187241 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187256 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187270 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187284 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdhh\" (UniqueName: \"kubernetes.io/projected/4bae7342-6a60-4145-8420-13c5bdf0beb0-kube-api-access-ccdhh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.191678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.187300 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.263729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.259612 5074 patch_prober.go:29] interesting pod/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 31 13:29:57.263729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.259796 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 31 13:29:57.307683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.301828 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-envoy" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:29:57.307683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.303547 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-maas-api-tls\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.307683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.303676 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9544b2c-8723-464e-88e1-bb656e81b058-openshift-service-ca\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.342690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.337721 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6nb\" (UniqueName: \"kubernetes.io/projected/b9544b2c-8723-464e-88e1-bb656e81b058-kube-api-access-vl6nb\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.342690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.337892 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-metrics-certs\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.342690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.338295 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.343863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.343822 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-token" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:57.363342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.361821 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-kube-api-access-c5ttt" pod "82ba5666-3275-407a-90b6-bcd8adb12e64" (UID: "82ba5666-3275-407a-90b6-bcd8adb12e64"). InnerVolumeSpecName "kube-api-access-c5ttt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:57.439581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.439544 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-maas-api-tls\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.439782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.439678 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9544b2c-8723-464e-88e1-bb656e81b058-openshift-service-ca\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.439782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.439728 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6nb\" (UniqueName: \"kubernetes.io/projected/b9544b2c-8723-464e-88e1-bb656e81b058-kube-api-access-vl6nb\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.439867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.439820 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-metrics-certs\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.440153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.440134 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ttt\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-kube-api-access-c5ttt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.440328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.440311 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba5666-3275-407a-90b6-bcd8adb12e64-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:57.440438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.440315 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9544b2c-8723-464e-88e1-bb656e81b058-openshift-service-ca\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.442337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.442314 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-maas-api-tls\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.442702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.442680 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-metrics-certs\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.454213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.454191 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6nb\" (UniqueName: \"kubernetes.io/projected/b9544b2c-8723-464e-88e1-bb656e81b058-kube-api-access-vl6nb\") pod \"maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.501214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.501125 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:57.619741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.619710 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg"] Oct 31 13:29:57.627735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.627712 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg"] Oct 31 13:29:57.628004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.627943 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.633796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.633389 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:29:57.633796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.633453 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:29:57.741838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.741801 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" Oct 31 13:29:57.742423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.741793 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk" event={"ID":"4bae7342-6a60-4145-8420-13c5bdf0beb0","Type":"ContainerDied","Data":"771d1018f13c5c9b1ea460843bb2b2217b24a785172ee183eaa907040c123702"} Oct 31 13:29:57.742508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.742471 5074 scope.go:122] "RemoveContainer" containerID="f4dd03090444c2ad107d1882dd0bd75faae4d22833e096ff10ff68d6ba0a12df" Oct 31 13:29:57.744697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.744570 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" Oct 31 13:29:57.744875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.744732 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh" event={"ID":"82ba5666-3275-407a-90b6-bcd8adb12e64","Type":"ContainerDied","Data":"1ba94bf847573afab97ef88367a4169c783b9244b4f635f9d3456df3043021ad"} Oct 31 13:29:57.745543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.745517 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.745723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.745700 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.745883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.745863 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26kgp\" (UniqueName: \"kubernetes.io/projected/296b3798-ffd7-4ca7-b409-b966928828f7-kube-api-access-26kgp\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.759148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.759128 5074 scope.go:122] "RemoveContainer" containerID="32ba2e750b3936d2830d5b9d43c289df5d2f95dbc39272b55c48564447a01555" Oct 31 13:29:57.830864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.830783 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk"] Oct 31 13:29:57.837804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.837775 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-nh9mk"] Oct 31 13:29:57.844148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.844091 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh"] Oct 31 13:29:57.848731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.847972 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.848731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.848057 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.848731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.848168 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26kgp\" (UniqueName: \"kubernetes.io/projected/296b3798-ffd7-4ca7-b409-b966928828f7-kube-api-access-26kgp\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.849028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.848737 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-fm7lh"] Oct 31 13:29:57.849469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.849426 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.852730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.852708 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.856769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.856751 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp"] Oct 31 13:29:57.861318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.861298 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26kgp\" (UniqueName: \"kubernetes.io/projected/296b3798-ffd7-4ca7-b409-b966928828f7-kube-api-access-26kgp\") pod \"maas-api-revoke-sub-92074962-sknrg\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:57.942515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.942481 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bae7342-6a60-4145-8420-13c5bdf0beb0" path="/var/lib/kubelet/pods/4bae7342-6a60-4145-8420-13c5bdf0beb0/volumes" Oct 31 13:29:57.943130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.943114 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ba5666-3275-407a-90b6-bcd8adb12e64" path="/var/lib/kubelet/pods/82ba5666-3275-407a-90b6-bcd8adb12e64/volumes" Oct 31 13:29:57.952208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:57.952182 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:29:58.223667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.213005 5074 patch_prober.go:29] interesting pod/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 31 13:29:58.239793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.239740 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 31 13:29:58.552605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.552573 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg"] Oct 31 13:29:58.553014 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:29:58.552987 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296b3798_ffd7_4ca7_b409_b966928828f7.slice/crio-52b81ac64119ffe50c1dcbcba50bdbea6e249298995cf933c506180964771c8a WatchSource:0}: Error finding container 52b81ac64119ffe50c1dcbcba50bdbea6e249298995cf933c506180964771c8a: Status 404 returned error can't find the container with id 52b81ac64119ffe50c1dcbcba50bdbea6e249298995cf933c506180964771c8a Oct 31 13:29:58.861668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.858038 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" event={"ID":"296b3798-ffd7-4ca7-b409-b966928828f7","Type":"ContainerStarted","Data":"52b81ac64119ffe50c1dcbcba50bdbea6e249298995cf933c506180964771c8a"} Oct 31 13:29:58.904918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.895137 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" event={"ID":"b9544b2c-8723-464e-88e1-bb656e81b058","Type":"ContainerStarted","Data":"0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e"} Oct 31 13:29:58.904918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.895212 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" event={"ID":"b9544b2c-8723-464e-88e1-bb656e81b058","Type":"ContainerStarted","Data":"6ef10e2523f08e599fb60f322d267a659cebc120a99e1194522db6056b2edd54"} Oct 31 13:29:58.904918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.896553 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:29:58.942602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:58.942542 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" podStartSLOduration=27.361031848 podStartE2EDuration="27.94252439s" totalImagesPullingTime="581.492542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:58.933005273 +0000 UTC m=+3493.110723257" watchObservedRunningTime="2026-10-31 13:29:58.94252439 +0000 UTC m=+3493.120242377" Oct 31 13:29:59.211575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:59.210888 5074 patch_prober.go:29] interesting pod/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 31 13:29:59.211575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:59.210999 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 31 13:29:59.954829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:29:59.954795 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" event={"ID":"296b3798-ffd7-4ca7-b409-b966928828f7","Type":"ContainerStarted","Data":"ca6e7d007d23aacc315a573df2c53ff94d776477b9a16649fc6a669b5e4d28b8"} Oct 31 13:30:00.165838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.165750 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs"] Oct 31 13:30:00.185765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.184911 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.214656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.211216 5074 patch_prober.go:29] interesting pod/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 31 13:30:00.214656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.211277 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 31 13:30:00.244683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.242578 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs"] Oct 31 13:30:00.270694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.269180 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k"] Oct 31 13:30:00.322657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.313970 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7v2s\" (UniqueName: \"kubernetes.io/projected/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-kube-api-access-t7v2s\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.322657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.316398 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.322657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.321878 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.329526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.326361 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r"] Oct 31 13:30:00.346966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.346941 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm"] Oct 31 13:30:00.347192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.347062 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.351717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.351697 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd"] Oct 31 13:30:00.351876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.351857 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.355933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.355912 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc"] Oct 31 13:30:00.356407 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.356378 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.360541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.360519 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k"] Oct 31 13:30:00.360678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.360625 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.363663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.363597 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm"] Oct 31 13:30:00.428051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.425491 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.428051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.425561 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.428051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.425723 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2zk\" (UniqueName: \"kubernetes.io/projected/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-kube-api-access-6d2zk\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.428051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.425906 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7v2s\" (UniqueName: \"kubernetes.io/projected/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-kube-api-access-t7v2s\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.428051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.427229 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.428051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.427460 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r"] Oct 31 13:30:00.439673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.435336 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd"] Oct 31 13:30:00.445885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.444746 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc"] Oct 31 13:30:00.449688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.447903 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85"] Oct 31 13:30:00.451786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.451762 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.456569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.456547 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7"] Oct 31 13:30:00.468225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.466795 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f"] Oct 31 13:30:00.468225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.466934 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.530426 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvjz\" (UniqueName: \"kubernetes.io/projected/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-kube-api-access-crvjz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.530599 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b118f65-c0a9-4a84-9f75-f76948b41c8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.530737 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zbw\" (UniqueName: \"kubernetes.io/projected/8b118f65-c0a9-4a84-9f75-f76948b41c8c-kube-api-access-m5zbw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.530789 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mt8f\" (UniqueName: \"kubernetes.io/projected/c7357272-2464-4f33-b723-597f306797bd-kube-api-access-8mt8f\") pod \"maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.530817 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6pg\" (UniqueName: \"kubernetes.io/projected/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-kube-api-access-sg6pg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.530872 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.530925 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7357272-2464-4f33-b723-597f306797bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.531026 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2zk\" (UniqueName: \"kubernetes.io/projected/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-kube-api-access-6d2zk\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.531055 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.531093 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.532431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.532063 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.537658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.534383 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7v2s\" (UniqueName: \"kubernetes.io/projected/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-kube-api-access-t7v2s\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.546498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.543736 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7"] Oct 31 13:30:00.546498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.543848 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:00.549117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.549096 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85"] Oct 31 13:30:00.556328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.555736 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f"] Oct 31 13:30:00.625556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.625380 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2zk\" (UniqueName: \"kubernetes.io/projected/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-kube-api-access-6d2zk\") pod \"maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.634128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.634093 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.634414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.634369 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.634709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.634688 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvjz\" (UniqueName: \"kubernetes.io/projected/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-kube-api-access-crvjz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.634915 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.634878 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:00.636777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.635340 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.636876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.636092 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.637016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.636987 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e2652-dde7-413e-97fd-c2ee731dbe67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.637152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.637138 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28htc\" (UniqueName: \"kubernetes.io/projected/646e2652-dde7-413e-97fd-c2ee731dbe67-kube-api-access-28htc\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.637278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.637265 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b118f65-c0a9-4a84-9f75-f76948b41c8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.637417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.637399 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2773a8c-6beb-4fe6-844e-200aebb724a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.637854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.637828 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zbw\" (UniqueName: \"kubernetes.io/projected/8b118f65-c0a9-4a84-9f75-f76948b41c8c-kube-api-access-m5zbw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.638029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.638001 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf65b\" (UniqueName: \"kubernetes.io/projected/e2773a8c-6beb-4fe6-844e-200aebb724a3-kube-api-access-tf65b\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.638185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.638167 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mt8f\" (UniqueName: \"kubernetes.io/projected/c7357272-2464-4f33-b723-597f306797bd-kube-api-access-8mt8f\") pod \"maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.638381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.638362 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6pg\" (UniqueName: \"kubernetes.io/projected/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-kube-api-access-sg6pg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.638490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.638034 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b118f65-c0a9-4a84-9f75-f76948b41c8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.638662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.638618 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7357272-2464-4f33-b723-597f306797bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.638821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.638802 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hzb\" (UniqueName: \"kubernetes.io/projected/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-kube-api-access-z7hzb\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:00.639333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.639311 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7357272-2464-4f33-b723-597f306797bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.648120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.648099 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:00.660173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.660144 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvjz\" (UniqueName: \"kubernetes.io/projected/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-kube-api-access-crvjz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.665822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.665795 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mt8f\" (UniqueName: \"kubernetes.io/projected/c7357272-2464-4f33-b723-597f306797bd-kube-api-access-8mt8f\") pod \"maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.665929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.665836 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6pg\" (UniqueName: \"kubernetes.io/projected/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-kube-api-access-sg6pg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.666543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.666517 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zbw\" (UniqueName: \"kubernetes.io/projected/8b118f65-c0a9-4a84-9f75-f76948b41c8c-kube-api-access-m5zbw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.668546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.668525 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:00.739020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.738967 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.743042 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.743170 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e2652-dde7-413e-97fd-c2ee731dbe67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.743227 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28htc\" (UniqueName: \"kubernetes.io/projected/646e2652-dde7-413e-97fd-c2ee731dbe67-kube-api-access-28htc\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.743271 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2773a8c-6beb-4fe6-844e-200aebb724a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.743374 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf65b\" (UniqueName: \"kubernetes.io/projected/e2773a8c-6beb-4fe6-844e-200aebb724a3-kube-api-access-tf65b\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.743475 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hzb\" (UniqueName: \"kubernetes.io/projected/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-kube-api-access-z7hzb\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.744466 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2773a8c-6beb-4fe6-844e-200aebb724a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.744574 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:00.744896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.744771 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e2652-dde7-413e-97fd-c2ee731dbe67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.799665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.784004 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:00.823993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.822286 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28htc\" (UniqueName: \"kubernetes.io/projected/646e2652-dde7-413e-97fd-c2ee731dbe67-kube-api-access-28htc\") pod \"maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.823993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.823578 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:00.831834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.830700 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hzb\" (UniqueName: \"kubernetes.io/projected/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-kube-api-access-z7hzb\") pod \"maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:00.831834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.830745 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf65b\" (UniqueName: \"kubernetes.io/projected/e2773a8c-6beb-4fe6-844e-200aebb724a3-kube-api-access-tf65b\") pod \"maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.845591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.845544 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:00.927654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.917272 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:00.927654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.927423 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:00.957716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:00.957565 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:01.044868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.044837 5074 generic.go:356] "Generic (PLEG): container finished" podID="296b3798-ffd7-4ca7-b409-b966928828f7" containerID="ca6e7d007d23aacc315a573df2c53ff94d776477b9a16649fc6a669b5e4d28b8" exitCode=0 Oct 31 13:30:01.045866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.045837 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" event={"ID":"296b3798-ffd7-4ca7-b409-b966928828f7","Type":"ContainerDied","Data":"ca6e7d007d23aacc315a573df2c53ff94d776477b9a16649fc6a669b5e4d28b8"} Oct 31 13:30:01.218719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.218401 5074 patch_prober.go:29] interesting pod/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Oct 31 13:30:01.218719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.218476 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Oct 31 13:30:01.661542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.661513 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:30:01.722675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.699265 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd"] Oct 31 13:30:01.722675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.700613 5074 cpu_manager.go:414] "containerMap: removing container" podUID="296b3798-ffd7-4ca7-b409-b966928828f7" containerName="revoke-keys" Oct 31 13:30:01.722675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.700653 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296b3798-ffd7-4ca7-b409-b966928828f7" containerName="revoke-keys" Oct 31 13:30:01.722675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.700960 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="296b3798-ffd7-4ca7-b409-b966928828f7" containerName="revoke-keys" Oct 31 13:30:01.723052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.722994 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:01.733819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.733772 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296b3798-ffd7-4ca7-b409-b966928828f7-kube-api-access-26kgp\" (UniqueName: \"kubernetes.io/projected/296b3798-ffd7-4ca7-b409-b966928828f7-kube-api-access-26kgp\") pod \"296b3798-ffd7-4ca7-b409-b966928828f7\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " Oct 31 13:30:01.733946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.733903 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-openshift-ca\") pod \"296b3798-ffd7-4ca7-b409-b966928828f7\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " Oct 31 13:30:01.734008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.733983 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-xks-ca\") pod \"296b3798-ffd7-4ca7-b409-b966928828f7\" (UID: \"296b3798-ffd7-4ca7-b409-b966928828f7\") " Oct 31 13:30:01.734535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.734398 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7pnw\" (UniqueName: \"kubernetes.io/projected/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-kube-api-access-x7pnw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:01.734535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.734439 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:01.737256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.737218 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd"] Oct 31 13:30:01.737653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.737578 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-openshift-ca" pod "296b3798-ffd7-4ca7-b409-b966928828f7" (UID: "296b3798-ffd7-4ca7-b409-b966928828f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:01.741104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.740893 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-xks-ca" pod "296b3798-ffd7-4ca7-b409-b966928828f7" (UID: "296b3798-ffd7-4ca7-b409-b966928828f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:01.741104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.740954 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296b3798-ffd7-4ca7-b409-b966928828f7-kube-api-access-26kgp" pod "296b3798-ffd7-4ca7-b409-b966928828f7" (UID: "296b3798-ffd7-4ca7-b409-b966928828f7"). InnerVolumeSpecName "kube-api-access-26kgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:01.742221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.742154 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k"] Oct 31 13:30:01.822065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.821820 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r"] Oct 31 13:30:01.835861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.835829 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7pnw\" (UniqueName: \"kubernetes.io/projected/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-kube-api-access-x7pnw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:01.835983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.835882 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:01.836180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.836159 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26kgp\" (UniqueName: \"kubernetes.io/projected/296b3798-ffd7-4ca7-b409-b966928828f7-kube-api-access-26kgp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:01.836283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.836187 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:01.836283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.836203 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296b3798-ffd7-4ca7-b409-b966928828f7-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:01.836826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.836801 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:01.850171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.850143 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7pnw\" (UniqueName: \"kubernetes.io/projected/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-kube-api-access-x7pnw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:01.913579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.913551 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm"] Oct 31 13:30:01.921237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.921208 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85"] Oct 31 13:30:01.992941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:01.985056 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd"] Oct 31 13:30:02.022470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.022434 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f"] Oct 31 13:30:02.058607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.058342 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:02.060687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.060368 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc"] Oct 31 13:30:02.066199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.066163 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerStarted","Data":"6df3eab750996056980686d4483f93b65280b0ed903dcbaf7d526cfe7c116614"} Oct 31 13:30:02.084346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.084289 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" event={"ID":"8b118f65-c0a9-4a84-9f75-f76948b41c8c","Type":"ContainerStarted","Data":"f5d61f00cab9aa2a4563eba3c6fa8e0cd133d821352bb4c735b9d6ce9f627986"} Oct 31 13:30:02.086200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.086148 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerStarted","Data":"521bb7d0d7fff5a057565d7e62960757949babe9812072a7b865aa76418a86bc"} Oct 31 13:30:02.088927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.088902 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" event={"ID":"3a4214bb-d6ed-44f0-bcdd-56c754d88f73","Type":"ContainerStarted","Data":"e4da24c29795fcbfd8bd6ec6f5664d23a71185379eb552c59171bd7a6447b313"} Oct 31 13:30:02.092071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.092042 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerStarted","Data":"a0e197f6aad61f7ede6243bb7cd40d76bcdbc1363b5389b6f164df3a22dab55d"} Oct 31 13:30:02.094105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.094028 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" event={"ID":"296b3798-ffd7-4ca7-b409-b966928828f7","Type":"ContainerDied","Data":"52b81ac64119ffe50c1dcbcba50bdbea6e249298995cf933c506180964771c8a"} Oct 31 13:30:02.094105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.094072 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52b81ac64119ffe50c1dcbcba50bdbea6e249298995cf933c506180964771c8a" Oct 31 13:30:02.094232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.094141 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg" Oct 31 13:30:02.100446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.100419 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerStarted","Data":"8632e4befc3ff22561819847700353748d2d4130740c7b285e23142f5b33f54c"} Oct 31 13:30:02.246788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.246282 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:30:02.253157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.248422 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:30:02.254954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.254868 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:30:02.259019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.258997 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs"] Oct 31 13:30:02.337058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.337012 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7"] Oct 31 13:30:02.751918 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:02.751874 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fa64c7_5eed_46d0_9771_8e5554ee2ac8.slice/crio-94e8674259aa16ed79b01fbeff7fa6120d2c4f8b832da0bee12d1094c21c3aa7 WatchSource:0}: Error finding container 94e8674259aa16ed79b01fbeff7fa6120d2c4f8b832da0bee12d1094c21c3aa7: Status 404 returned error can't find the container with id 94e8674259aa16ed79b01fbeff7fa6120d2c4f8b832da0bee12d1094c21c3aa7 Oct 31 13:30:02.752179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:02.752151 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd"] Oct 31 13:30:03.118247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.118205 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" event={"ID":"646e2652-dde7-413e-97fd-c2ee731dbe67","Type":"ContainerStarted","Data":"a903a3611107369cabbb713bbb5b78c177111b503db8e77c3c10ccf60d5ce651"} Oct 31 13:30:03.118751 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.118277 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" event={"ID":"646e2652-dde7-413e-97fd-c2ee731dbe67","Type":"ContainerStarted","Data":"b56a59f72ee3094d50f48d1d40ee2d73d928211f63ec054d6bf68fdeed4e63dc"} Oct 31 13:30:03.125476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.125432 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerStarted","Data":"3a4046c0ddb804ce203d328eccfe95b8f3ea7d9b5613f115817bd4886095f3e3"} Oct 31 13:30:03.128477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.128288 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" event={"ID":"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97","Type":"ContainerStarted","Data":"1fbeb95796c2c8c32e5f83a06ccdba6f66d85492ddc80123bd138b1635b2e507"} Oct 31 13:30:03.128477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.128343 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" event={"ID":"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97","Type":"ContainerStarted","Data":"f3a040fbd80c09d79da2621e422d8a8f9f5f410a175e2f29b6bdf1fa4e2990d4"} Oct 31 13:30:03.132520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.132248 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerStarted","Data":"9ef1d434c77ecee0cf714759afe253cd93cf6d6011c527974574c89ea980363b"} Oct 31 13:30:03.134362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.134330 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerStarted","Data":"ddb688a2b9efc3e65a7a23ac346971dd4789b9420791adca313f19c464984843"} Oct 31 13:30:03.137363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.136866 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerStarted","Data":"94e8674259aa16ed79b01fbeff7fa6120d2c4f8b832da0bee12d1094c21c3aa7"} Oct 31 13:30:03.139432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.139404 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" event={"ID":"8b118f65-c0a9-4a84-9f75-f76948b41c8c","Type":"ContainerStarted","Data":"581b217a78d1d8488a29f605f0b9c5c4ece242476894fcc4bce639b98a3fcd83"} Oct 31 13:30:03.143081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.143053 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerStarted","Data":"8839e1441e05b125c0e0f6b1d5a98681987d9b2c68597fa6f72512d2ad3f1c20"} Oct 31 13:30:03.146391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.145470 5074 generic.go:356] "Generic (PLEG): container finished" podID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerID="99e31d2d811c270c5d854256604b68c6dc61d58f726421edd7727477277e6de7" exitCode=60 Oct 31 13:30:03.146391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.145562 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" event={"ID":"3a4214bb-d6ed-44f0-bcdd-56c754d88f73","Type":"ContainerDied","Data":"99e31d2d811c270c5d854256604b68c6dc61d58f726421edd7727477277e6de7"} Oct 31 13:30:03.146563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.146538 5074 scope.go:122] "RemoveContainer" containerID="99e31d2d811c270c5d854256604b68c6dc61d58f726421edd7727477277e6de7" Oct 31 13:30:03.149129 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.149106 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" event={"ID":"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d","Type":"ContainerStarted","Data":"640b0ce0646f266466b7a9b8830c3aedd46dd6c1e0df0b0b00f90b5541246756"} Oct 31 13:30:03.149214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.149148 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" event={"ID":"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d","Type":"ContainerStarted","Data":"cf0017a8e349a00575f6f1c983fef83bb0e763e47418826171e5a90d1225fc33"} Oct 31 13:30:03.158110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.158069 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" podStartSLOduration=3.158058087 podStartE2EDuration="3.158058087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.157663558 +0000 UTC m=+3497.335381538" watchObservedRunningTime="2026-10-31 13:30:03.158058087 +0000 UTC m=+3497.335776071" Oct 31 13:30:03.162536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.162496 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" podStartSLOduration=3.162484068 podStartE2EDuration="3.162484068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.138291221 +0000 UTC m=+3497.316009209" watchObservedRunningTime="2026-10-31 13:30:03.162484068 +0000 UTC m=+3497.340202051" Oct 31 13:30:03.196424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.196360 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" podStartSLOduration=3.196340749 podStartE2EDuration="3.196340749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.192842996 +0000 UTC m=+3497.370560981" watchObservedRunningTime="2026-10-31 13:30:03.196340749 +0000 UTC m=+3497.374058734" Oct 31 13:30:03.244682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.227628 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" podStartSLOduration=3.227610218 podStartE2EDuration="3.227610218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.215333031 +0000 UTC m=+3497.393051029" watchObservedRunningTime="2026-10-31 13:30:03.227610218 +0000 UTC m=+3497.405328205" Oct 31 13:30:03.251965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.251901 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" podStartSLOduration=3.251878769 podStartE2EDuration="3.251878769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.245949564 +0000 UTC m=+3497.423667549" watchObservedRunningTime="2026-10-31 13:30:03.251878769 +0000 UTC m=+3497.429596789" Oct 31 13:30:03.434974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.417145 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" podStartSLOduration=3.417125364 podStartE2EDuration="3.417125364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.294289347 +0000 UTC m=+3497.472007331" watchObservedRunningTime="2026-10-31 13:30:03.417125364 +0000 UTC m=+3497.594843349" Oct 31 13:30:03.670426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.669808 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" podStartSLOduration=3.669790074 podStartE2EDuration="3.669790074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.363332096 +0000 UTC m=+3497.541050084" watchObservedRunningTime="2026-10-31 13:30:03.669790074 +0000 UTC m=+3497.847508059" Oct 31 13:30:03.696301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:03.696230 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" podStartSLOduration=3.696210041 podStartE2EDuration="3.696210041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:03.416554863 +0000 UTC m=+3497.594272848" watchObservedRunningTime="2026-10-31 13:30:03.696210041 +0000 UTC m=+3497.873928029" Oct 31 13:30:04.165852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.165810 5074 generic.go:356] "Generic (PLEG): container finished" podID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerID="640b0ce0646f266466b7a9b8830c3aedd46dd6c1e0df0b0b00f90b5541246756" exitCode=60 Oct 31 13:30:04.166346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.166012 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" event={"ID":"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d","Type":"ContainerDied","Data":"640b0ce0646f266466b7a9b8830c3aedd46dd6c1e0df0b0b00f90b5541246756"} Oct 31 13:30:04.166346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.166306 5074 scope.go:122] "RemoveContainer" containerID="640b0ce0646f266466b7a9b8830c3aedd46dd6c1e0df0b0b00f90b5541246756" Oct 31 13:30:04.169187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.169164 5074 generic.go:356] "Generic (PLEG): container finished" podID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerID="a903a3611107369cabbb713bbb5b78c177111b503db8e77c3c10ccf60d5ce651" exitCode=60 Oct 31 13:30:04.169323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.169299 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" event={"ID":"646e2652-dde7-413e-97fd-c2ee731dbe67","Type":"ContainerDied","Data":"a903a3611107369cabbb713bbb5b78c177111b503db8e77c3c10ccf60d5ce651"} Oct 31 13:30:04.169805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.169785 5074 scope.go:122] "RemoveContainer" containerID="a903a3611107369cabbb713bbb5b78c177111b503db8e77c3c10ccf60d5ce651" Oct 31 13:30:04.188213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.188188 5074 generic.go:356] "Generic (PLEG): container finished" podID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerID="3a4046c0ddb804ce203d328eccfe95b8f3ea7d9b5613f115817bd4886095f3e3" exitCode=60 Oct 31 13:30:04.188985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.188967 5074 scope.go:122] "RemoveContainer" containerID="3a4046c0ddb804ce203d328eccfe95b8f3ea7d9b5613f115817bd4886095f3e3" Oct 31 13:30:04.189485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.189463 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerDied","Data":"3a4046c0ddb804ce203d328eccfe95b8f3ea7d9b5613f115817bd4886095f3e3"} Oct 31 13:30:04.196923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.196896 5074 generic.go:356] "Generic (PLEG): container finished" podID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerID="1fbeb95796c2c8c32e5f83a06ccdba6f66d85492ddc80123bd138b1635b2e507" exitCode=60 Oct 31 13:30:04.197040 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.197014 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" event={"ID":"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97","Type":"ContainerDied","Data":"1fbeb95796c2c8c32e5f83a06ccdba6f66d85492ddc80123bd138b1635b2e507"} Oct 31 13:30:04.197167 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.197150 5074 scope.go:122] "RemoveContainer" containerID="1fbeb95796c2c8c32e5f83a06ccdba6f66d85492ddc80123bd138b1635b2e507" Oct 31 13:30:04.206865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.206839 5074 generic.go:356] "Generic (PLEG): container finished" podID="c7357272-2464-4f33-b723-597f306797bd" containerID="9ef1d434c77ecee0cf714759afe253cd93cf6d6011c527974574c89ea980363b" exitCode=60 Oct 31 13:30:04.206964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.206947 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerDied","Data":"9ef1d434c77ecee0cf714759afe253cd93cf6d6011c527974574c89ea980363b"} Oct 31 13:30:04.207352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.207323 5074 scope.go:122] "RemoveContainer" containerID="9ef1d434c77ecee0cf714759afe253cd93cf6d6011c527974574c89ea980363b" Oct 31 13:30:04.229744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.227852 5074 generic.go:356] "Generic (PLEG): container finished" podID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerID="ddb688a2b9efc3e65a7a23ac346971dd4789b9420791adca313f19c464984843" exitCode=60 Oct 31 13:30:04.229744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.227951 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerDied","Data":"ddb688a2b9efc3e65a7a23ac346971dd4789b9420791adca313f19c464984843"} Oct 31 13:30:04.229744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.229589 5074 scope.go:122] "RemoveContainer" containerID="ddb688a2b9efc3e65a7a23ac346971dd4789b9420791adca313f19c464984843" Oct 31 13:30:04.238411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.235898 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerStarted","Data":"1c3a8e54b5f59431543f1d3efb6ec111437fe688ec8dbccc77d50fa30806a3d2"} Oct 31 13:30:04.248222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.248195 5074 generic.go:356] "Generic (PLEG): container finished" podID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerID="581b217a78d1d8488a29f605f0b9c5c4ece242476894fcc4bce639b98a3fcd83" exitCode=60 Oct 31 13:30:04.248464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.248441 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" event={"ID":"8b118f65-c0a9-4a84-9f75-f76948b41c8c","Type":"ContainerDied","Data":"581b217a78d1d8488a29f605f0b9c5c4ece242476894fcc4bce639b98a3fcd83"} Oct 31 13:30:04.248950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.248935 5074 scope.go:122] "RemoveContainer" containerID="581b217a78d1d8488a29f605f0b9c5c4ece242476894fcc4bce639b98a3fcd83" Oct 31 13:30:04.251609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.251583 5074 generic.go:356] "Generic (PLEG): container finished" podID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerID="8839e1441e05b125c0e0f6b1d5a98681987d9b2c68597fa6f72512d2ad3f1c20" exitCode=60 Oct 31 13:30:04.252773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.252708 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerDied","Data":"8839e1441e05b125c0e0f6b1d5a98681987d9b2c68597fa6f72512d2ad3f1c20"} Oct 31 13:30:04.252880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.252845 5074 scope.go:122] "RemoveContainer" containerID="8839e1441e05b125c0e0f6b1d5a98681987d9b2c68597fa6f72512d2ad3f1c20" Oct 31 13:30:04.339395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:04.339337 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" podStartSLOduration=4.339314913 podStartE2EDuration="4.339314913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:04.318932208 +0000 UTC m=+3498.496650192" watchObservedRunningTime="2026-10-31 13:30:04.339314913 +0000 UTC m=+3498.517032903" Oct 31 13:30:05.287624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.287509 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerStarted","Data":"15a8ae600b6d8d09def7f2f0932e6078424faf7dce80e2251891b6cbc2194ce5"} Oct 31 13:30:05.291538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.291502 5074 generic.go:356] "Generic (PLEG): container finished" podID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerID="498b55736e94f454783cb2bc670ae526eb851bb12fe8f74e54ec5e332ce65972" exitCode=60 Oct 31 13:30:05.291757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.291625 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" event={"ID":"3a4214bb-d6ed-44f0-bcdd-56c754d88f73","Type":"ContainerDied","Data":"498b55736e94f454783cb2bc670ae526eb851bb12fe8f74e54ec5e332ce65972"} Oct 31 13:30:05.291757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.291732 5074 scope.go:122] "RemoveContainer" containerID="99e31d2d811c270c5d854256604b68c6dc61d58f726421edd7727477277e6de7" Oct 31 13:30:05.291977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.291952 5074 scope.go:122] "RemoveContainer" containerID="498b55736e94f454783cb2bc670ae526eb851bb12fe8f74e54ec5e332ce65972" Oct 31 13:30:05.292552 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:05.292526 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k_odh-ai-gateway-infra(3a4214bb-d6ed-44f0-bcdd-56c754d88f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" Oct 31 13:30:05.294976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.294953 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" event={"ID":"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d","Type":"ContainerStarted","Data":"04956a9f5f7f3c434b4d30201a74512392d9c80f96306441f89cd8e195fa2475"} Oct 31 13:30:05.297570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.297546 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" event={"ID":"646e2652-dde7-413e-97fd-c2ee731dbe67","Type":"ContainerStarted","Data":"d591f65cc608cd713136105ec3bf22bd13ad514d43bf236806bf2db7e6466ebf"} Oct 31 13:30:05.300449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.300426 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerStarted","Data":"11ef2cdb10bfa41fcf3fc64dcd258e8ca8684fce4427a24f1427a02622296c39"} Oct 31 13:30:05.302812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.302790 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" event={"ID":"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97","Type":"ContainerStarted","Data":"97c5feb9a9b793f0c53bf60d0d9e94fa8dbf2717daf3658a01e0fe6fe83c64ee"} Oct 31 13:30:05.304558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.304536 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerStarted","Data":"7840ac59912424fac68fde3f52b5e764a459752d5e952e72a730c1bc83de9df6"} Oct 31 13:30:05.313237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.313209 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerStarted","Data":"cef952a622b7809d3e6ad7fcabb7a60d8f7b4a7fd76a16482a44ce3afa7b829d"} Oct 31 13:30:05.317272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.316533 5074 generic.go:356] "Generic (PLEG): container finished" podID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerID="1c3a8e54b5f59431543f1d3efb6ec111437fe688ec8dbccc77d50fa30806a3d2" exitCode=60 Oct 31 13:30:05.317272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.316614 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerDied","Data":"1c3a8e54b5f59431543f1d3efb6ec111437fe688ec8dbccc77d50fa30806a3d2"} Oct 31 13:30:05.317272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.316925 5074 scope.go:122] "RemoveContainer" containerID="1c3a8e54b5f59431543f1d3efb6ec111437fe688ec8dbccc77d50fa30806a3d2" Oct 31 13:30:05.321991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:05.321968 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" event={"ID":"8b118f65-c0a9-4a84-9f75-f76948b41c8c","Type":"ContainerStarted","Data":"864f728ff3cc79e2388abf71f9f497c8defa8edf4c035bbbf859a05487e3c5d5"} Oct 31 13:30:06.051311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.051284 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:30:06.335189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.335150 5074 generic.go:356] "Generic (PLEG): container finished" podID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerID="15a8ae600b6d8d09def7f2f0932e6078424faf7dce80e2251891b6cbc2194ce5" exitCode=60 Oct 31 13:30:06.335672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.335231 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerDied","Data":"15a8ae600b6d8d09def7f2f0932e6078424faf7dce80e2251891b6cbc2194ce5"} Oct 31 13:30:06.335672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.335294 5074 scope.go:122] "RemoveContainer" containerID="8839e1441e05b125c0e0f6b1d5a98681987d9b2c68597fa6f72512d2ad3f1c20" Oct 31 13:30:06.335839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.335685 5074 scope.go:122] "RemoveContainer" containerID="15a8ae600b6d8d09def7f2f0932e6078424faf7dce80e2251891b6cbc2194ce5" Oct 31 13:30:06.336144 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.336115 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f_odh-ai-gateway-infra(6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" Oct 31 13:30:06.337548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.337524 5074 scope.go:122] "RemoveContainer" containerID="498b55736e94f454783cb2bc670ae526eb851bb12fe8f74e54ec5e332ce65972" Oct 31 13:30:06.337992 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.337957 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k_odh-ai-gateway-infra(3a4214bb-d6ed-44f0-bcdd-56c754d88f73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" Oct 31 13:30:06.339321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.339299 5074 generic.go:356] "Generic (PLEG): container finished" podID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerID="04956a9f5f7f3c434b4d30201a74512392d9c80f96306441f89cd8e195fa2475" exitCode=60 Oct 31 13:30:06.339677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.339505 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" event={"ID":"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d","Type":"ContainerDied","Data":"04956a9f5f7f3c434b4d30201a74512392d9c80f96306441f89cd8e195fa2475"} Oct 31 13:30:06.339787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.339743 5074 scope.go:122] "RemoveContainer" containerID="04956a9f5f7f3c434b4d30201a74512392d9c80f96306441f89cd8e195fa2475" Oct 31 13:30:06.340102 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.340078 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs_odh-ai-gateway-infra(3c6e3708-5e4a-49f1-8849-cb4bd983bc1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" Oct 31 13:30:06.341692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.341669 5074 generic.go:356] "Generic (PLEG): container finished" podID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerID="d591f65cc608cd713136105ec3bf22bd13ad514d43bf236806bf2db7e6466ebf" exitCode=60 Oct 31 13:30:06.341770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.341754 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" event={"ID":"646e2652-dde7-413e-97fd-c2ee731dbe67","Type":"ContainerDied","Data":"d591f65cc608cd713136105ec3bf22bd13ad514d43bf236806bf2db7e6466ebf"} Oct 31 13:30:06.342022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.341996 5074 scope.go:122] "RemoveContainer" containerID="d591f65cc608cd713136105ec3bf22bd13ad514d43bf236806bf2db7e6466ebf" Oct 31 13:30:06.342364 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.342340 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7_odh-ai-gateway-infra(646e2652-dde7-413e-97fd-c2ee731dbe67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" Oct 31 13:30:06.344003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.343980 5074 generic.go:356] "Generic (PLEG): container finished" podID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerID="11ef2cdb10bfa41fcf3fc64dcd258e8ca8684fce4427a24f1427a02622296c39" exitCode=60 Oct 31 13:30:06.344095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.344062 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerDied","Data":"11ef2cdb10bfa41fcf3fc64dcd258e8ca8684fce4427a24f1427a02622296c39"} Oct 31 13:30:06.344322 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.344301 5074 scope.go:122] "RemoveContainer" containerID="11ef2cdb10bfa41fcf3fc64dcd258e8ca8684fce4427a24f1427a02622296c39" Oct 31 13:30:06.344696 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.344618 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85_odh-ai-gateway-infra(e2773a8c-6beb-4fe6-844e-200aebb724a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" Oct 31 13:30:06.347067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.347046 5074 generic.go:356] "Generic (PLEG): container finished" podID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerID="97c5feb9a9b793f0c53bf60d0d9e94fa8dbf2717daf3658a01e0fe6fe83c64ee" exitCode=60 Oct 31 13:30:06.347273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.347243 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" event={"ID":"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97","Type":"ContainerDied","Data":"97c5feb9a9b793f0c53bf60d0d9e94fa8dbf2717daf3658a01e0fe6fe83c64ee"} Oct 31 13:30:06.347741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.347719 5074 scope.go:122] "RemoveContainer" containerID="97c5feb9a9b793f0c53bf60d0d9e94fa8dbf2717daf3658a01e0fe6fe83c64ee" Oct 31 13:30:06.348005 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.347981 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc_odh-ai-gateway-infra(a83865ad-ffe7-47f5-8a1d-fb923e0a6b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" Oct 31 13:30:06.349909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.349883 5074 generic.go:356] "Generic (PLEG): container finished" podID="c7357272-2464-4f33-b723-597f306797bd" containerID="7840ac59912424fac68fde3f52b5e764a459752d5e952e72a730c1bc83de9df6" exitCode=60 Oct 31 13:30:06.350025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.349980 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerDied","Data":"7840ac59912424fac68fde3f52b5e764a459752d5e952e72a730c1bc83de9df6"} Oct 31 13:30:06.350346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.350325 5074 scope.go:122] "RemoveContainer" containerID="7840ac59912424fac68fde3f52b5e764a459752d5e952e72a730c1bc83de9df6" Oct 31 13:30:06.351093 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.350606 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r_odh-ai-gateway-infra(c7357272-2464-4f33-b723-597f306797bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" podUID="c7357272-2464-4f33-b723-597f306797bd" Oct 31 13:30:06.352289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.352265 5074 generic.go:356] "Generic (PLEG): container finished" podID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerID="cef952a622b7809d3e6ad7fcabb7a60d8f7b4a7fd76a16482a44ce3afa7b829d" exitCode=60 Oct 31 13:30:06.352378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.352349 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerDied","Data":"cef952a622b7809d3e6ad7fcabb7a60d8f7b4a7fd76a16482a44ce3afa7b829d"} Oct 31 13:30:06.352596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.352574 5074 scope.go:122] "RemoveContainer" containerID="cef952a622b7809d3e6ad7fcabb7a60d8f7b4a7fd76a16482a44ce3afa7b829d" Oct 31 13:30:06.352716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.352679 5074 scope.go:122] "RemoveContainer" containerID="640b0ce0646f266466b7a9b8830c3aedd46dd6c1e0df0b0b00f90b5541246756" Oct 31 13:30:06.352953 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.352928 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm_odh-ai-gateway-infra(ffa79e5f-4f76-4cbb-be95-50a999cb3ccb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" Oct 31 13:30:06.354435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.354408 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerStarted","Data":"d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb"} Oct 31 13:30:06.359094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.359035 5074 generic.go:356] "Generic (PLEG): container finished" podID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerID="864f728ff3cc79e2388abf71f9f497c8defa8edf4c035bbbf859a05487e3c5d5" exitCode=60 Oct 31 13:30:06.359094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.359088 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" event={"ID":"8b118f65-c0a9-4a84-9f75-f76948b41c8c","Type":"ContainerDied","Data":"864f728ff3cc79e2388abf71f9f497c8defa8edf4c035bbbf859a05487e3c5d5"} Oct 31 13:30:06.359429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.359410 5074 scope.go:122] "RemoveContainer" containerID="864f728ff3cc79e2388abf71f9f497c8defa8edf4c035bbbf859a05487e3c5d5" Oct 31 13:30:06.359808 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.359783 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd_odh-ai-gateway-infra(8b118f65-c0a9-4a84-9f75-f76948b41c8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" Oct 31 13:30:06.375865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.368755 5074 scope.go:122] "RemoveContainer" containerID="a903a3611107369cabbb713bbb5b78c177111b503db8e77c3c10ccf60d5ce651" Oct 31 13:30:06.384601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.384581 5074 scope.go:122] "RemoveContainer" containerID="3a4046c0ddb804ce203d328eccfe95b8f3ea7d9b5613f115817bd4886095f3e3" Oct 31 13:30:06.404320 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.404246 5074 scope.go:122] "RemoveContainer" containerID="1fbeb95796c2c8c32e5f83a06ccdba6f66d85492ddc80123bd138b1635b2e507" Oct 31 13:30:06.430338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.430304 5074 scope.go:122] "RemoveContainer" containerID="9ef1d434c77ecee0cf714759afe253cd93cf6d6011c527974574c89ea980363b" Oct 31 13:30:06.445207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.445186 5074 scope.go:122] "RemoveContainer" containerID="ddb688a2b9efc3e65a7a23ac346971dd4789b9420791adca313f19c464984843" Oct 31 13:30:06.458520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:06.458496 5074 scope.go:122] "RemoveContainer" containerID="581b217a78d1d8488a29f605f0b9c5c4ece242476894fcc4bce639b98a3fcd83" Oct 31 13:30:06.908143 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:06.908100 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fa64c7_5eed_46d0_9771_8e5554ee2ac8.slice/crio-d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fa64c7_5eed_46d0_9771_8e5554ee2ac8.slice/crio-conmon-d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:30:07.371122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.371083 5074 scope.go:122] "RemoveContainer" containerID="04956a9f5f7f3c434b4d30201a74512392d9c80f96306441f89cd8e195fa2475" Oct 31 13:30:07.371622 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.371545 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs_odh-ai-gateway-infra(3c6e3708-5e4a-49f1-8849-cb4bd983bc1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" Oct 31 13:30:07.373230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.373203 5074 scope.go:122] "RemoveContainer" containerID="d591f65cc608cd713136105ec3bf22bd13ad514d43bf236806bf2db7e6466ebf" Oct 31 13:30:07.373857 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.373826 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7_odh-ai-gateway-infra(646e2652-dde7-413e-97fd-c2ee731dbe67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" Oct 31 13:30:07.375320 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.375295 5074 scope.go:122] "RemoveContainer" containerID="11ef2cdb10bfa41fcf3fc64dcd258e8ca8684fce4427a24f1427a02622296c39" Oct 31 13:30:07.375562 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.375541 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85_odh-ai-gateway-infra(e2773a8c-6beb-4fe6-844e-200aebb724a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" Oct 31 13:30:07.377766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.377739 5074 scope.go:122] "RemoveContainer" containerID="97c5feb9a9b793f0c53bf60d0d9e94fa8dbf2717daf3658a01e0fe6fe83c64ee" Oct 31 13:30:07.378054 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.378026 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc_odh-ai-gateway-infra(a83865ad-ffe7-47f5-8a1d-fb923e0a6b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" Oct 31 13:30:07.379443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.379424 5074 scope.go:122] "RemoveContainer" containerID="7840ac59912424fac68fde3f52b5e764a459752d5e952e72a730c1bc83de9df6" Oct 31 13:30:07.379731 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.379703 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r_odh-ai-gateway-infra(c7357272-2464-4f33-b723-597f306797bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" podUID="c7357272-2464-4f33-b723-597f306797bd" Oct 31 13:30:07.381522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.381503 5074 scope.go:122] "RemoveContainer" containerID="cef952a622b7809d3e6ad7fcabb7a60d8f7b4a7fd76a16482a44ce3afa7b829d" Oct 31 13:30:07.381780 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.381761 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm_odh-ai-gateway-infra(ffa79e5f-4f76-4cbb-be95-50a999cb3ccb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" Oct 31 13:30:07.383459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.383434 5074 generic.go:356] "Generic (PLEG): container finished" podID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerID="d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb" exitCode=60 Oct 31 13:30:07.383540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.383512 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerDied","Data":"d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb"} Oct 31 13:30:07.383615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.383573 5074 scope.go:122] "RemoveContainer" containerID="1c3a8e54b5f59431543f1d3efb6ec111437fe688ec8dbccc77d50fa30806a3d2" Oct 31 13:30:07.383751 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.383733 5074 scope.go:122] "RemoveContainer" containerID="d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb" Oct 31 13:30:07.383983 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.383965 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd_odh-ai-gateway-infra(17fa64c7-5eed-46d0-9771-8e5554ee2ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" Oct 31 13:30:07.385933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.385915 5074 scope.go:122] "RemoveContainer" containerID="864f728ff3cc79e2388abf71f9f497c8defa8edf4c035bbbf859a05487e3c5d5" Oct 31 13:30:07.386209 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.386181 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd_odh-ai-gateway-infra(8b118f65-c0a9-4a84-9f75-f76948b41c8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" Oct 31 13:30:07.387853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:07.387835 5074 scope.go:122] "RemoveContainer" containerID="15a8ae600b6d8d09def7f2f0932e6078424faf7dce80e2251891b6cbc2194ce5" Oct 31 13:30:07.388072 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:07.388053 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f_odh-ai-gateway-infra(6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" Oct 31 13:30:08.446852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:08.444680 5074 scope.go:122] "RemoveContainer" containerID="d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb" Oct 31 13:30:08.446852 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:08.445173 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd_odh-ai-gateway-infra(17fa64c7-5eed-46d0-9771-8e5554ee2ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" Oct 31 13:30:09.631656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:09.626791 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws"] Oct 31 13:30:10.269615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:10.269578 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:30:14.627208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:14.627142 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" podUID="0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" containerName="payload-processing" containerID="cri-o://17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde" gracePeriod=25 Oct 31 13:30:14.965191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:14.965163 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:30:15.060604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.060568 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-kube-api-access-ztwjh\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-kube-api-access-ztwjh\") pod \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " Oct 31 13:30:15.060916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.060894 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-provider-credentials\") pod \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " Oct 31 13:30:15.061099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.061083 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-plugins-config-volume\") pod \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " Oct 31 13:30:15.061523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.061504 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-routing-overlay-volume\") pod \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\" (UID: \"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11\") " Oct 31 13:30:15.062517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.062468 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-plugins-config-volume" pod "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" (UID: "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:15.062901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.062879 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:15.064343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.064313 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-routing-overlay-volume" pod "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" (UID: "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:15.076311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.076283 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9"] Oct 31 13:30:15.077547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.077515 5074 cpu_manager.go:414] "containerMap: removing container" podUID="0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" containerName="payload-processing" Oct 31 13:30:15.077547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.077546 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" containerName="payload-processing" Oct 31 13:30:15.078832 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.077802 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" containerName="payload-processing" Oct 31 13:30:15.081612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.080887 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-kube-api-access-ztwjh" pod "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" (UID: "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11"). InnerVolumeSpecName "kube-api-access-ztwjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:15.081612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.081025 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-provider-credentials" pod "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" (UID: "0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:15.082351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.082327 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.083833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.083807 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg"] Oct 31 13:30:15.110689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.110660 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.123679 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.123654 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb"] Oct 31 13:30:15.135209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.135062 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.135397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.134948 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw"] Oct 31 13:30:15.146481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.146454 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg"] Oct 31 13:30:15.147912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.147888 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.151608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.151588 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.154986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.154966 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg"] Oct 31 13:30:15.160110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.160090 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9"] Oct 31 13:30:15.164174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164146 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98z9n\" (UniqueName: \"kubernetes.io/projected/086f29d4-9e94-45fd-a9cb-d36f19841d44-kube-api-access-98z9n\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.164285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164216 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xmm\" (UniqueName: \"kubernetes.io/projected/6e89d8ad-db41-4694-8db6-524a9c06496f-kube-api-access-r9xmm\") pod \"maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.164285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164266 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e89d8ad-db41-4694-8db6-524a9c06496f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.164418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164344 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.164418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164406 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396c4960-19d8-499d-95ce-3e5f995f07fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.164523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164463 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.164575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164529 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5w2d\" (UniqueName: \"kubernetes.io/projected/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-kube-api-access-d5w2d\") pod \"maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.164664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164612 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4974\" (UniqueName: \"kubernetes.io/projected/396c4960-19d8-499d-95ce-3e5f995f07fc-kube-api-access-b4974\") pod \"maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.164852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164830 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fs2\" (UniqueName: \"kubernetes.io/projected/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-kube-api-access-f6fs2\") pod \"maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.164931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.164914 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086f29d4-9e94-45fd-a9cb-d36f19841d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.165112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.165092 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztwjh\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-kube-api-access-ztwjh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:15.165209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.165120 5074 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-provider-credentials\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:15.165209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.165135 5074 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11-routing-overlay-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:15.167189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.167161 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg"] Oct 31 13:30:15.177328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.177275 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw"] Oct 31 13:30:15.183681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.183606 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb"] Oct 31 13:30:15.266480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266448 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.266677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266516 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5w2d\" (UniqueName: \"kubernetes.io/projected/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-kube-api-access-d5w2d\") pod \"maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.266677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266582 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4974\" (UniqueName: \"kubernetes.io/projected/396c4960-19d8-499d-95ce-3e5f995f07fc-kube-api-access-b4974\") pod \"maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.266843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266729 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fs2\" (UniqueName: \"kubernetes.io/projected/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-kube-api-access-f6fs2\") pod \"maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.266843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266796 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086f29d4-9e94-45fd-a9cb-d36f19841d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.266990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266865 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98z9n\" (UniqueName: \"kubernetes.io/projected/086f29d4-9e94-45fd-a9cb-d36f19841d44-kube-api-access-98z9n\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.266990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266919 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xmm\" (UniqueName: \"kubernetes.io/projected/6e89d8ad-db41-4694-8db6-524a9c06496f-kube-api-access-r9xmm\") pod \"maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.266990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.266967 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e89d8ad-db41-4694-8db6-524a9c06496f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.267179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.267035 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.267179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.267101 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396c4960-19d8-499d-95ce-3e5f995f07fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.267361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.267332 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.267620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.267591 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086f29d4-9e94-45fd-a9cb-d36f19841d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.267841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.267797 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e89d8ad-db41-4694-8db6-524a9c06496f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.267941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.267837 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396c4960-19d8-499d-95ce-3e5f995f07fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.268025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.268004 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.279704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.279675 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fs2\" (UniqueName: \"kubernetes.io/projected/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-kube-api-access-f6fs2\") pod \"maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.280134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.280106 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4974\" (UniqueName: \"kubernetes.io/projected/396c4960-19d8-499d-95ce-3e5f995f07fc-kube-api-access-b4974\") pod \"maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.280388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.280366 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5w2d\" (UniqueName: \"kubernetes.io/projected/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-kube-api-access-d5w2d\") pod \"maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.280667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.280615 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98z9n\" (UniqueName: \"kubernetes.io/projected/086f29d4-9e94-45fd-a9cb-d36f19841d44-kube-api-access-98z9n\") pod \"maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.280829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.280808 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xmm\" (UniqueName: \"kubernetes.io/projected/6e89d8ad-db41-4694-8db6-524a9c06496f-kube-api-access-r9xmm\") pod \"maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.454906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.454808 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:15.469698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.469666 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:15.483695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.483669 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:15.490477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.490396 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:15.498176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.498155 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:15.601447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.601263 5074 generic.go:356] "Generic (PLEG): container finished" podID="0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" containerID="17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde" exitCode=0 Oct 31 13:30:15.601934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.601900 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" event={"ID":"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11","Type":"ContainerDied","Data":"17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde"} Oct 31 13:30:15.602062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.601967 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" event={"ID":"0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11","Type":"ContainerDied","Data":"0a374173aa4ddb3154d8ffef203bdb3978e4f0d8c76fce405d0d74174d4d5f4d"} Oct 31 13:30:15.602062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.602002 5074 scope.go:122] "RemoveContainer" containerID="17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde" Oct 31 13:30:15.602222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.602206 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws" Oct 31 13:30:15.624764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.624611 5074 scope.go:122] "RemoveContainer" containerID="17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde" Oct 31 13:30:15.625239 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:15.625176 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde\": container with ID starting with 17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde not found: ID does not exist" containerID="17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde" Oct 31 13:30:15.625239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.625207 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde"} err="failed to get container status \"17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde\": rpc error: code = NotFound desc = could not find container \"17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde\": container with ID starting with 17cbbfe80e0a8f02e13645f14dbcc8942163ce6867d2d427f84d3593f64dccde not found: ID does not exist" Oct 31 13:30:15.653941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.653882 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws"] Oct 31 13:30:15.659091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.659066 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5a9e45/payload-processing-external-model-e2e-extmod-b-5a9e45-644dcldws"] Oct 31 13:30:15.773300 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:15.772839 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396c4960_19d8_499d_95ce_3e5f995f07fc.slice/crio-1615dca522560993cc9a849b4c4baad10014fc8d1f96e0a34a5e820daad0940d WatchSource:0}: Error finding container 1615dca522560993cc9a849b4c4baad10014fc8d1f96e0a34a5e820daad0940d: Status 404 returned error can't find the container with id 1615dca522560993cc9a849b4c4baad10014fc8d1f96e0a34a5e820daad0940d Oct 31 13:30:15.773300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.772985 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg"] Oct 31 13:30:15.775689 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:15.775621 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e89d8ad_db41_4694_8db6_524a9c06496f.slice/crio-7942cc2c74ff51b15ddc58f27a92ed3dff99daeb7a2681b7785f1a98b3ce7224 WatchSource:0}: Error finding container 7942cc2c74ff51b15ddc58f27a92ed3dff99daeb7a2681b7785f1a98b3ce7224: Status 404 returned error can't find the container with id 7942cc2c74ff51b15ddc58f27a92ed3dff99daeb7a2681b7785f1a98b3ce7224 Oct 31 13:30:15.780747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.780721 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9"] Oct 31 13:30:15.860114 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:15.860081 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2c78e2_3449_4bef_86cd_2d09c24af2ac.slice/crio-c0a268373dd2a558f73e52a5d6799e3f8e31384ff49e148a8d7cacb526994729 WatchSource:0}: Error finding container c0a268373dd2a558f73e52a5d6799e3f8e31384ff49e148a8d7cacb526994729: Status 404 returned error can't find the container with id c0a268373dd2a558f73e52a5d6799e3f8e31384ff49e148a8d7cacb526994729 Oct 31 13:30:15.861725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.861688 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg"] Oct 31 13:30:15.940004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.939977 5074 scope.go:122] "RemoveContainer" containerID="11ef2cdb10bfa41fcf3fc64dcd258e8ca8684fce4427a24f1427a02622296c39" Oct 31 13:30:15.946461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:15.946435 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11" path="/var/lib/kubelet/pods/0b68ad0e-fe50-4a9f-a7ac-3ab9f813ee11/volumes" Oct 31 13:30:16.056723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.052704 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb"] Oct 31 13:30:16.092256 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:16.090818 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fbf28a_3926_4559_aea4_b9e6b5c5c2e3.slice/crio-a8677ee72ddcd98bfa1e296fca9d0a820f363aa7142c35d001bfa38315ddf15f WatchSource:0}: Error finding container a8677ee72ddcd98bfa1e296fca9d0a820f363aa7142c35d001bfa38315ddf15f: Status 404 returned error can't find the container with id a8677ee72ddcd98bfa1e296fca9d0a820f363aa7142c35d001bfa38315ddf15f Oct 31 13:30:16.092256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.091384 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z"] Oct 31 13:30:16.100539 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.100500 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeaf5726-brz8z"] Oct 31 13:30:16.111811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.111741 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw"] Oct 31 13:30:16.612523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.612490 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerStarted","Data":"f95a4c68e028c1233da13521fa19ac2ae397a81bf5b47e0bf6ce6452cf1cafce"} Oct 31 13:30:16.614491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.614457 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerStarted","Data":"70a14e498b0466723fadfe7ed877e05ff1524ff375b86a2c4caee35eba71e59a"} Oct 31 13:30:16.614612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.614515 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerStarted","Data":"1615dca522560993cc9a849b4c4baad10014fc8d1f96e0a34a5e820daad0940d"} Oct 31 13:30:16.616069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.616037 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" event={"ID":"086f29d4-9e94-45fd-a9cb-d36f19841d44","Type":"ContainerStarted","Data":"913715d25f8350588bec14ee95849617ca74ba487f817bf59148405632d6b338"} Oct 31 13:30:16.617823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.617790 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" event={"ID":"6e89d8ad-db41-4694-8db6-524a9c06496f","Type":"ContainerStarted","Data":"46924d45e1be575768a81a10d19a498a0c9506231cfd854c91faf25df64db4fa"} Oct 31 13:30:16.617914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.617860 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" event={"ID":"6e89d8ad-db41-4694-8db6-524a9c06496f","Type":"ContainerStarted","Data":"7942cc2c74ff51b15ddc58f27a92ed3dff99daeb7a2681b7785f1a98b3ce7224"} Oct 31 13:30:16.619479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.619454 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" event={"ID":"9a2c78e2-3449-4bef-86cd-2d09c24af2ac","Type":"ContainerStarted","Data":"2fde9cfdd8e32b52ffbdba45cc8d50c685fa4776d30426ead7d9ad270b153482"} Oct 31 13:30:16.619565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.619503 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" event={"ID":"9a2c78e2-3449-4bef-86cd-2d09c24af2ac","Type":"ContainerStarted","Data":"c0a268373dd2a558f73e52a5d6799e3f8e31384ff49e148a8d7cacb526994729"} Oct 31 13:30:16.621091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.621062 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" event={"ID":"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3","Type":"ContainerStarted","Data":"a8677ee72ddcd98bfa1e296fca9d0a820f363aa7142c35d001bfa38315ddf15f"} Oct 31 13:30:16.658169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.658080 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" podStartSLOduration=16.658066749 podStartE2EDuration="16.658066749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:16.651257205 +0000 UTC m=+3510.828975192" watchObservedRunningTime="2026-10-31 13:30:16.658066749 +0000 UTC m=+3510.835784732" Oct 31 13:30:16.698868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.698806 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" podStartSLOduration=16.698787812 podStartE2EDuration="16.698787812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:16.675263223 +0000 UTC m=+3510.852981236" watchObservedRunningTime="2026-10-31 13:30:16.698787812 +0000 UTC m=+3510.876505797" Oct 31 13:30:16.701925 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.701878 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" podStartSLOduration=16.701866768 podStartE2EDuration="16.701866768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:16.697071659 +0000 UTC m=+3510.874789643" watchObservedRunningTime="2026-10-31 13:30:16.701866768 +0000 UTC m=+3510.879584791" Oct 31 13:30:16.931707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.930949 5074 scope.go:122] "RemoveContainer" containerID="498b55736e94f454783cb2bc670ae526eb851bb12fe8f74e54ec5e332ce65972" Oct 31 13:30:16.931707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.931233 5074 scope.go:122] "RemoveContainer" containerID="864f728ff3cc79e2388abf71f9f497c8defa8edf4c035bbbf859a05487e3c5d5" Oct 31 13:30:16.931945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:16.931892 5074 scope.go:122] "RemoveContainer" containerID="04956a9f5f7f3c434b4d30201a74512392d9c80f96306441f89cd8e195fa2475" Oct 31 13:30:17.635786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.635746 5074 generic.go:356] "Generic (PLEG): container finished" podID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerID="46924d45e1be575768a81a10d19a498a0c9506231cfd854c91faf25df64db4fa" exitCode=60 Oct 31 13:30:17.635965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.635840 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" event={"ID":"6e89d8ad-db41-4694-8db6-524a9c06496f","Type":"ContainerDied","Data":"46924d45e1be575768a81a10d19a498a0c9506231cfd854c91faf25df64db4fa"} Oct 31 13:30:17.636141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.636123 5074 scope.go:122] "RemoveContainer" containerID="46924d45e1be575768a81a10d19a498a0c9506231cfd854c91faf25df64db4fa" Oct 31 13:30:17.638302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.638187 5074 generic.go:356] "Generic (PLEG): container finished" podID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerID="2fde9cfdd8e32b52ffbdba45cc8d50c685fa4776d30426ead7d9ad270b153482" exitCode=60 Oct 31 13:30:17.638455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.638427 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" event={"ID":"9a2c78e2-3449-4bef-86cd-2d09c24af2ac","Type":"ContainerDied","Data":"2fde9cfdd8e32b52ffbdba45cc8d50c685fa4776d30426ead7d9ad270b153482"} Oct 31 13:30:17.638772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.638746 5074 scope.go:122] "RemoveContainer" containerID="2fde9cfdd8e32b52ffbdba45cc8d50c685fa4776d30426ead7d9ad270b153482" Oct 31 13:30:17.640715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.640690 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" event={"ID":"3a4214bb-d6ed-44f0-bcdd-56c754d88f73","Type":"ContainerStarted","Data":"5af07f9c34e0bcfe24a34e08af10ba73190a66d5d93e7be6e96d4dc68b15e1b2"} Oct 31 13:30:17.642749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.642729 5074 generic.go:356] "Generic (PLEG): container finished" podID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerID="bb999090a4a6b82b4315b2c6ff1229495084102c8d904f60421b2f35328da49b" exitCode=60 Oct 31 13:30:17.643039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.642931 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" event={"ID":"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3","Type":"ContainerDied","Data":"bb999090a4a6b82b4315b2c6ff1229495084102c8d904f60421b2f35328da49b"} Oct 31 13:30:17.643329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.643310 5074 scope.go:122] "RemoveContainer" containerID="bb999090a4a6b82b4315b2c6ff1229495084102c8d904f60421b2f35328da49b" Oct 31 13:30:17.645223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.645198 5074 generic.go:356] "Generic (PLEG): container finished" podID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerID="f95a4c68e028c1233da13521fa19ac2ae397a81bf5b47e0bf6ce6452cf1cafce" exitCode=60 Oct 31 13:30:17.645321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.645306 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerDied","Data":"f95a4c68e028c1233da13521fa19ac2ae397a81bf5b47e0bf6ce6452cf1cafce"} Oct 31 13:30:17.645397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.645353 5074 scope.go:122] "RemoveContainer" containerID="11ef2cdb10bfa41fcf3fc64dcd258e8ca8684fce4427a24f1427a02622296c39" Oct 31 13:30:17.645786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.645765 5074 scope.go:122] "RemoveContainer" containerID="f95a4c68e028c1233da13521fa19ac2ae397a81bf5b47e0bf6ce6452cf1cafce" Oct 31 13:30:17.646112 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:17.646088 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85_odh-ai-gateway-infra(e2773a8c-6beb-4fe6-844e-200aebb724a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" Oct 31 13:30:17.648094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.648073 5074 generic.go:356] "Generic (PLEG): container finished" podID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerID="70a14e498b0466723fadfe7ed877e05ff1524ff375b86a2c4caee35eba71e59a" exitCode=60 Oct 31 13:30:17.648184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.648165 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerDied","Data":"70a14e498b0466723fadfe7ed877e05ff1524ff375b86a2c4caee35eba71e59a"} Oct 31 13:30:17.648446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.648430 5074 scope.go:122] "RemoveContainer" containerID="70a14e498b0466723fadfe7ed877e05ff1524ff375b86a2c4caee35eba71e59a" Oct 31 13:30:17.650408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.650383 5074 generic.go:356] "Generic (PLEG): container finished" podID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerID="a76d87ce240559dc635a3e2abd544a09c154799d02ff76f2e26b395a6c924cea" exitCode=60 Oct 31 13:30:17.650517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.650428 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" event={"ID":"086f29d4-9e94-45fd-a9cb-d36f19841d44","Type":"ContainerDied","Data":"a76d87ce240559dc635a3e2abd544a09c154799d02ff76f2e26b395a6c924cea"} Oct 31 13:30:17.650710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.650689 5074 scope.go:122] "RemoveContainer" containerID="a76d87ce240559dc635a3e2abd544a09c154799d02ff76f2e26b395a6c924cea" Oct 31 13:30:17.662928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.662605 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85"] Oct 31 13:30:17.734025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.733951 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" podStartSLOduration=17.733936969 podStartE2EDuration="17.733936969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:17.730586978 +0000 UTC m=+3511.908304980" watchObservedRunningTime="2026-10-31 13:30:17.733936969 +0000 UTC m=+3511.911654953" Oct 31 13:30:17.931723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.931688 5074 scope.go:122] "RemoveContainer" containerID="d591f65cc608cd713136105ec3bf22bd13ad514d43bf236806bf2db7e6466ebf" Oct 31 13:30:17.933955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.933924 5074 scope.go:122] "RemoveContainer" containerID="97c5feb9a9b793f0c53bf60d0d9e94fa8dbf2717daf3658a01e0fe6fe83c64ee" Oct 31 13:30:17.934872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.934504 5074 scope.go:122] "RemoveContainer" containerID="15a8ae600b6d8d09def7f2f0932e6078424faf7dce80e2251891b6cbc2194ce5" Oct 31 13:30:17.950197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:17.950171 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac975c0-e17a-4818-bbc0-8dc9e4702c4c" path="/var/lib/kubelet/pods/cac975c0-e17a-4818-bbc0-8dc9e4702c4c/volumes" Oct 31 13:30:18.788252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.784710 5074 generic.go:356] "Generic (PLEG): container finished" podID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerID="8418e476cf1defbc0e185f9674d9f2422be9185cc444a9c17403790b410b169e" exitCode=60 Oct 31 13:30:18.788252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.784967 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" event={"ID":"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d","Type":"ContainerDied","Data":"8418e476cf1defbc0e185f9674d9f2422be9185cc444a9c17403790b410b169e"} Oct 31 13:30:18.788252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.785029 5074 scope.go:122] "RemoveContainer" containerID="04956a9f5f7f3c434b4d30201a74512392d9c80f96306441f89cd8e195fa2475" Oct 31 13:30:18.788252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.785618 5074 scope.go:122] "RemoveContainer" containerID="8418e476cf1defbc0e185f9674d9f2422be9185cc444a9c17403790b410b169e" Oct 31 13:30:18.797149 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:18.797087 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs_odh-ai-gateway-infra(3c6e3708-5e4a-49f1-8849-cb4bd983bc1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" Oct 31 13:30:18.814128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.809087 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k"] Oct 31 13:30:18.814128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.811910 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerStarted","Data":"1bb368f973b63dbf107d7da35f75c8183185e91a997b3e895e75154985642ae2"} Oct 31 13:30:18.937029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.936942 5074 generic.go:356] "Generic (PLEG): container finished" podID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerID="d24df5a9114233c8544b6ba24f65979841e51b784cdd2f28c33c13f97f56f9d2" exitCode=60 Oct 31 13:30:18.937334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.937312 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" event={"ID":"8b118f65-c0a9-4a84-9f75-f76948b41c8c","Type":"ContainerDied","Data":"d24df5a9114233c8544b6ba24f65979841e51b784cdd2f28c33c13f97f56f9d2"} Oct 31 13:30:18.938042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.938012 5074 scope.go:122] "RemoveContainer" containerID="d24df5a9114233c8544b6ba24f65979841e51b784cdd2f28c33c13f97f56f9d2" Oct 31 13:30:18.938718 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:18.938693 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd_odh-ai-gateway-infra(8b118f65-c0a9-4a84-9f75-f76948b41c8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" Oct 31 13:30:18.981770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.977251 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerStarted","Data":"b8ce9c4d2850ec3f536f07e3e162ac8a49c50130c23ddb16737e394c30de49f2"} Oct 31 13:30:18.985615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.985587 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" event={"ID":"9a2c78e2-3449-4bef-86cd-2d09c24af2ac","Type":"ContainerStarted","Data":"383a2c6556482cb3d5af886de527abc853e81aa62aaeae61975e3d6f6c0ebcef"} Oct 31 13:30:18.993964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.989591 5074 generic.go:356] "Generic (PLEG): container finished" podID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerID="5af07f9c34e0bcfe24a34e08af10ba73190a66d5d93e7be6e96d4dc68b15e1b2" exitCode=60 Oct 31 13:30:18.993964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:18.989643 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" event={"ID":"3a4214bb-d6ed-44f0-bcdd-56c754d88f73","Type":"ContainerDied","Data":"5af07f9c34e0bcfe24a34e08af10ba73190a66d5d93e7be6e96d4dc68b15e1b2"} Oct 31 13:30:19.006075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.006046 5074 scope.go:122] "RemoveContainer" containerID="864f728ff3cc79e2388abf71f9f497c8defa8edf4c035bbbf859a05487e3c5d5" Oct 31 13:30:19.062848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.058297 5074 scope.go:122] "RemoveContainer" containerID="498b55736e94f454783cb2bc670ae526eb851bb12fe8f74e54ec5e332ce65972" Oct 31 13:30:19.107535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.098804 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" podStartSLOduration=19.098787155 podStartE2EDuration="19.098787155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:18.954610764 +0000 UTC m=+3513.132328749" watchObservedRunningTime="2026-10-31 13:30:19.098787155 +0000 UTC m=+3513.276505140" Oct 31 13:30:19.584456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.584428 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:19.628381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.624156 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg"] Oct 31 13:30:19.628381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.625464 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.628381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.625487 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.628381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.625560 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.628381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.625568 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.628381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.625814 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.628381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.625831 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.631419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.631392 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:19.634340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.634191 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:19.639694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.639671 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg"] Oct 31 13:30:19.739399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.739360 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2"] Oct 31 13:30:19.743136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.743106 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.743326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.743310 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.743537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.743524 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.743655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.743624 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.743772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.743760 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.743856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.743846 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.743940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.743926 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.744012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.744002 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.744393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.744371 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.744541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.744529 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.750761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.750739 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" containerName="cleanup" Oct 31 13:30:19.750865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.750786 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" containerName="cleanup" Oct 31 13:30:19.843756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.841994 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2773a8c-6beb-4fe6-844e-200aebb724a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2773a8c-6beb-4fe6-844e-200aebb724a3-maas-api-service-ca\") pod \"e2773a8c-6beb-4fe6-844e-200aebb724a3\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " Oct 31 13:30:19.843756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.842184 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-maas-api-service-ca\") pod \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " Oct 31 13:30:19.843756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.842284 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-kube-api-access-6d2zk\" (UniqueName: \"kubernetes.io/projected/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-kube-api-access-6d2zk\") pod \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\" (UID: \"3a4214bb-d6ed-44f0-bcdd-56c754d88f73\") " Oct 31 13:30:19.843756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.842359 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2773a8c-6beb-4fe6-844e-200aebb724a3-kube-api-access-tf65b\" (UniqueName: \"kubernetes.io/projected/e2773a8c-6beb-4fe6-844e-200aebb724a3-kube-api-access-tf65b\") pod \"e2773a8c-6beb-4fe6-844e-200aebb724a3\" (UID: \"e2773a8c-6beb-4fe6-844e-200aebb724a3\") " Oct 31 13:30:19.847689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.846608 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2773a8c-6beb-4fe6-844e-200aebb724a3-maas-api-service-ca" pod "e2773a8c-6beb-4fe6-844e-200aebb724a3" (UID: "e2773a8c-6beb-4fe6-844e-200aebb724a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:19.847689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.846620 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9mm\" (UniqueName: \"kubernetes.io/projected/3d7495d8-6556-4de7-aac1-d560a8b63fb0-kube-api-access-fk9mm\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:19.847689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.847086 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-maas-api-service-ca" pod "3a4214bb-d6ed-44f0-bcdd-56c754d88f73" (UID: "3a4214bb-d6ed-44f0-bcdd-56c754d88f73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:19.850033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.848110 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7495d8-6556-4de7-aac1-d560a8b63fb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:19.850033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.848495 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2773a8c-6beb-4fe6-844e-200aebb724a3-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:19.850033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.848515 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:19.853069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.853033 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:19.864942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.864901 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2773a8c-6beb-4fe6-844e-200aebb724a3-kube-api-access-tf65b" pod "e2773a8c-6beb-4fe6-844e-200aebb724a3" (UID: "e2773a8c-6beb-4fe6-844e-200aebb724a3"). InnerVolumeSpecName "kube-api-access-tf65b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:19.878762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.878722 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2"] Oct 31 13:30:19.889540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.889453 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs"] Oct 31 13:30:19.943667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.937427 5074 scope.go:122] "RemoveContainer" containerID="7840ac59912424fac68fde3f52b5e764a459752d5e952e72a730c1bc83de9df6" Oct 31 13:30:19.943667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.938028 5074 scope.go:122] "RemoveContainer" containerID="cef952a622b7809d3e6ad7fcabb7a60d8f7b4a7fd76a16482a44ce3afa7b829d" Oct 31 13:30:19.952268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.950726 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7495d8-6556-4de7-aac1-d560a8b63fb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:19.952268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.950982 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9mm\" (UniqueName: \"kubernetes.io/projected/3d7495d8-6556-4de7-aac1-d560a8b63fb0-kube-api-access-fk9mm\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:19.952268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.951090 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf65b\" (UniqueName: \"kubernetes.io/projected/e2773a8c-6beb-4fe6-844e-200aebb724a3-kube-api-access-tf65b\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:19.952268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.952089 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7495d8-6556-4de7-aac1-d560a8b63fb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:19.972350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.971224 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-kube-api-access-6d2zk" pod "3a4214bb-d6ed-44f0-bcdd-56c754d88f73" (UID: "3a4214bb-d6ed-44f0-bcdd-56c754d88f73"). InnerVolumeSpecName "kube-api-access-6d2zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:19.972350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:19.971536 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9mm\" (UniqueName: \"kubernetes.io/projected/3d7495d8-6556-4de7-aac1-d560a8b63fb0-kube-api-access-fk9mm\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:20.017417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.017385 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd"] Oct 31 13:30:20.028693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.028043 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" Oct 31 13:30:20.028693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.028042 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85" event={"ID":"e2773a8c-6beb-4fe6-844e-200aebb724a3","Type":"ContainerDied","Data":"a0e197f6aad61f7ede6243bb7cd40d76bcdbc1363b5389b6f164df3a22dab55d"} Oct 31 13:30:20.028693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.028188 5074 scope.go:122] "RemoveContainer" containerID="f95a4c68e028c1233da13521fa19ac2ae397a81bf5b47e0bf6ce6452cf1cafce" Oct 31 13:30:20.032979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.032949 5074 generic.go:356] "Generic (PLEG): container finished" podID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerID="a5f7ff532be40643c30494a2b2ee8b72e8035f4cbe3270008926443ecb20f38c" exitCode=60 Oct 31 13:30:20.033099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.033061 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" event={"ID":"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97","Type":"ContainerDied","Data":"a5f7ff532be40643c30494a2b2ee8b72e8035f4cbe3270008926443ecb20f38c"} Oct 31 13:30:20.033608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.033582 5074 scope.go:122] "RemoveContainer" containerID="a5f7ff532be40643c30494a2b2ee8b72e8035f4cbe3270008926443ecb20f38c" Oct 31 13:30:20.034099 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:20.034067 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc_odh-ai-gateway-infra(a83865ad-ffe7-47f5-8a1d-fb923e0a6b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" Oct 31 13:30:20.036012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.035985 5074 generic.go:356] "Generic (PLEG): container finished" podID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerID="1bb368f973b63dbf107d7da35f75c8183185e91a997b3e895e75154985642ae2" exitCode=60 Oct 31 13:30:20.036324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.036297 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerDied","Data":"1bb368f973b63dbf107d7da35f75c8183185e91a997b3e895e75154985642ae2"} Oct 31 13:30:20.037447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.037428 5074 scope.go:122] "RemoveContainer" containerID="1bb368f973b63dbf107d7da35f75c8183185e91a997b3e895e75154985642ae2" Oct 31 13:30:20.038263 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:20.038237 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg_odh-ai-gateway-infra(396c4960-19d8-499d-95ce-3e5f995f07fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" Oct 31 13:30:20.046747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.046721 5074 generic.go:356] "Generic (PLEG): container finished" podID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerID="a9d8ffd97317cc4a4d59282b3798dc668465ecd6c92e3e3d83c5758eb71128f7" exitCode=60 Oct 31 13:30:20.046881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.046822 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" event={"ID":"086f29d4-9e94-45fd-a9cb-d36f19841d44","Type":"ContainerDied","Data":"a9d8ffd97317cc4a4d59282b3798dc668465ecd6c92e3e3d83c5758eb71128f7"} Oct 31 13:30:20.047349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.047274 5074 scope.go:122] "RemoveContainer" containerID="a9d8ffd97317cc4a4d59282b3798dc668465ecd6c92e3e3d83c5758eb71128f7" Oct 31 13:30:20.047717 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:20.047690 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw_odh-ai-gateway-infra(086f29d4-9e94-45fd-a9cb-d36f19841d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" Oct 31 13:30:20.050376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.050356 5074 scope.go:122] "RemoveContainer" containerID="97c5feb9a9b793f0c53bf60d0d9e94fa8dbf2717daf3658a01e0fe6fe83c64ee" Oct 31 13:30:20.050956 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.050929 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f"] Oct 31 13:30:20.052559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.052532 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a23e0a1-17ab-434b-aa91-1ce660969ec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:20.052980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.052950 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9b7w\" (UniqueName: \"kubernetes.io/projected/5a23e0a1-17ab-434b-aa91-1ce660969ec0-kube-api-access-c9b7w\") pod \"maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:20.054458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.054433 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2zk\" (UniqueName: \"kubernetes.io/projected/3a4214bb-d6ed-44f0-bcdd-56c754d88f73-kube-api-access-6d2zk\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:20.055856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.055834 5074 generic.go:356] "Generic (PLEG): container finished" podID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerID="e4f1836c436edff037c7e023bc68381c7ad6a289df45f0bb6333e8210be1d968" exitCode=60 Oct 31 13:30:20.056044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.055911 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" event={"ID":"6e89d8ad-db41-4694-8db6-524a9c06496f","Type":"ContainerDied","Data":"e4f1836c436edff037c7e023bc68381c7ad6a289df45f0bb6333e8210be1d968"} Oct 31 13:30:20.056204 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.056185 5074 scope.go:122] "RemoveContainer" containerID="e4f1836c436edff037c7e023bc68381c7ad6a289df45f0bb6333e8210be1d968" Oct 31 13:30:20.056449 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:20.056430 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9_odh-ai-gateway-infra(6e89d8ad-db41-4694-8db6-524a9c06496f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" Oct 31 13:30:20.062734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.062694 5074 generic.go:356] "Generic (PLEG): container finished" podID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerID="b8ce9c4d2850ec3f536f07e3e162ac8a49c50130c23ddb16737e394c30de49f2" exitCode=60 Oct 31 13:30:20.062967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.062925 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerDied","Data":"b8ce9c4d2850ec3f536f07e3e162ac8a49c50130c23ddb16737e394c30de49f2"} Oct 31 13:30:20.065340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.065319 5074 generic.go:356] "Generic (PLEG): container finished" podID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerID="383a2c6556482cb3d5af886de527abc853e81aa62aaeae61975e3d6f6c0ebcef" exitCode=60 Oct 31 13:30:20.065443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.065429 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" event={"ID":"9a2c78e2-3449-4bef-86cd-2d09c24af2ac","Type":"ContainerDied","Data":"383a2c6556482cb3d5af886de527abc853e81aa62aaeae61975e3d6f6c0ebcef"} Oct 31 13:30:20.065869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.065846 5074 scope.go:122] "RemoveContainer" containerID="383a2c6556482cb3d5af886de527abc853e81aa62aaeae61975e3d6f6c0ebcef" Oct 31 13:30:20.066268 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:20.066243 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg_odh-ai-gateway-infra(9a2c78e2-3449-4bef-86cd-2d09c24af2ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" Oct 31 13:30:20.068334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.068311 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" event={"ID":"3a4214bb-d6ed-44f0-bcdd-56c754d88f73","Type":"ContainerDied","Data":"e4da24c29795fcbfd8bd6ec6f5664d23a71185379eb552c59171bd7a6447b313"} Oct 31 13:30:20.068431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.068392 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k" Oct 31 13:30:20.082504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.081045 5074 scope.go:122] "RemoveContainer" containerID="70a14e498b0466723fadfe7ed877e05ff1524ff375b86a2c4caee35eba71e59a" Oct 31 13:30:20.114692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.114622 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85"] Oct 31 13:30:20.118128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.118094 5074 generic.go:356] "Generic (PLEG): container finished" podID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerID="a49f1583ee44512e729208d64807433eec87f0c3941c7c8104399fa397e70f92" exitCode=60 Oct 31 13:30:20.118557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.118295 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" event={"ID":"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3","Type":"ContainerDied","Data":"a49f1583ee44512e729208d64807433eec87f0c3941c7c8104399fa397e70f92"} Oct 31 13:30:20.119248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.119225 5074 scope.go:122] "RemoveContainer" containerID="a49f1583ee44512e729208d64807433eec87f0c3941c7c8104399fa397e70f92" Oct 31 13:30:20.120091 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:20.119851 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb_odh-ai-gateway-infra(22fbf28a-3926-4559-aea4-b9e6b5c5c2e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" Oct 31 13:30:20.124683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.122981 5074 generic.go:356] "Generic (PLEG): container finished" podID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerID="6fc6907cfdfdb5546947c57702184ce4d170bb14b66e1bfb0b73163281aa1224" exitCode=60 Oct 31 13:30:20.124683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.123039 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" event={"ID":"646e2652-dde7-413e-97fd-c2ee731dbe67","Type":"ContainerDied","Data":"6fc6907cfdfdb5546947c57702184ce4d170bb14b66e1bfb0b73163281aa1224"} Oct 31 13:30:20.124683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.123515 5074 scope.go:122] "RemoveContainer" containerID="6fc6907cfdfdb5546947c57702184ce4d170bb14b66e1bfb0b73163281aa1224" Oct 31 13:30:20.124683 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:20.124003 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7_odh-ai-gateway-infra(646e2652-dde7-413e-97fd-c2ee731dbe67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" Oct 31 13:30:20.135141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.135122 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-29e3a0-29890890-9dw85"] Oct 31 13:30:20.142518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.142196 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:20.157109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.157076 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9b7w\" (UniqueName: \"kubernetes.io/projected/5a23e0a1-17ab-434b-aa91-1ce660969ec0-kube-api-access-c9b7w\") pod \"maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:20.157574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.157545 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a23e0a1-17ab-434b-aa91-1ce660969ec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:20.158528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.158500 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a23e0a1-17ab-434b-aa91-1ce660969ec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:20.171534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.170853 5074 scope.go:122] "RemoveContainer" containerID="a76d87ce240559dc635a3e2abd544a09c154799d02ff76f2e26b395a6c924cea" Oct 31 13:30:20.227723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.213994 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9b7w\" (UniqueName: \"kubernetes.io/projected/5a23e0a1-17ab-434b-aa91-1ce660969ec0-kube-api-access-c9b7w\") pod \"maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:20.243793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.242694 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:20.251048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.250986 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k"] Oct 31 13:30:20.264828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.264706 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-54a16b-29890890-k2r8k"] Oct 31 13:30:20.279015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.272492 5074 scope.go:122] "RemoveContainer" containerID="46924d45e1be575768a81a10d19a498a0c9506231cfd854c91faf25df64db4fa" Oct 31 13:30:20.419309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.418181 5074 scope.go:122] "RemoveContainer" containerID="15a8ae600b6d8d09def7f2f0932e6078424faf7dce80e2251891b6cbc2194ce5" Oct 31 13:30:20.448225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.448195 5074 scope.go:122] "RemoveContainer" containerID="2fde9cfdd8e32b52ffbdba45cc8d50c685fa4776d30426ead7d9ad270b153482" Oct 31 13:30:20.464829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.464800 5074 scope.go:122] "RemoveContainer" containerID="5af07f9c34e0bcfe24a34e08af10ba73190a66d5d93e7be6e96d4dc68b15e1b2" Oct 31 13:30:20.539479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.539449 5074 scope.go:122] "RemoveContainer" containerID="bb999090a4a6b82b4315b2c6ff1229495084102c8d904f60421b2f35328da49b" Oct 31 13:30:20.572159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.566545 5074 scope.go:122] "RemoveContainer" containerID="d591f65cc608cd713136105ec3bf22bd13ad514d43bf236806bf2db7e6466ebf" Oct 31 13:30:20.619828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.619798 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:20.663519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.663464 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq"] Oct 31 13:30:20.665203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.665148 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.665390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.665371 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.665570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.665556 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.665772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.665726 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.665963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.665947 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.666122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.666106 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.666556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.666538 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.666715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.666703 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.685492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.685471 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq"] Oct 31 13:30:20.685604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.685580 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:20.708686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.708658 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b118f65-c0a9-4a84-9f75-f76948b41c8c-maas-api-service-ca" pod "8b118f65-c0a9-4a84-9f75-f76948b41c8c" (UID: "8b118f65-c0a9-4a84-9f75-f76948b41c8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:20.712689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.709460 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b118f65-c0a9-4a84-9f75-f76948b41c8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b118f65-c0a9-4a84-9f75-f76948b41c8c-maas-api-service-ca\") pod \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " Oct 31 13:30:20.712689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.710419 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b118f65-c0a9-4a84-9f75-f76948b41c8c-kube-api-access-m5zbw\" (UniqueName: \"kubernetes.io/projected/8b118f65-c0a9-4a84-9f75-f76948b41c8c-kube-api-access-m5zbw\") pod \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\" (UID: \"8b118f65-c0a9-4a84-9f75-f76948b41c8c\") " Oct 31 13:30:20.713340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.713308 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b118f65-c0a9-4a84-9f75-f76948b41c8c-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:20.715056 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.715017 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b118f65-c0a9-4a84-9f75-f76948b41c8c-kube-api-access-m5zbw" pod "8b118f65-c0a9-4a84-9f75-f76948b41c8c" (UID: "8b118f65-c0a9-4a84-9f75-f76948b41c8c"). InnerVolumeSpecName "kube-api-access-m5zbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:20.780601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.780522 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:20.786771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.786747 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:20.903687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.852934 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2nv\" (UniqueName: \"kubernetes.io/projected/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-kube-api-access-gx2nv\") pod \"maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:20.903687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.897959 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:20.903687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.898374 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5zbw\" (UniqueName: \"kubernetes.io/projected/8b118f65-c0a9-4a84-9f75-f76948b41c8c-kube-api-access-m5zbw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:20.909106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.909075 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb"] Oct 31 13:30:20.910710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.910683 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.910860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.910846 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.911006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.910994 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.911091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911080 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.911173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911164 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.911245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911235 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.911360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911314 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.911446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911433 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.911554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911543 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.911657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911646 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.911744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911734 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.911828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.911818 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.912244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.912226 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.912353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.912342 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.912437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.912428 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" containerName="cleanup" Oct 31 13:30:20.912550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.912539 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:20.912623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.912614 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:20.919184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.919165 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg"] Oct 31 13:30:20.920155 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.919283 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:20.926508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.926486 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb"] Oct 31 13:30:20.936120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.936097 5074 scope.go:122] "RemoveContainer" containerID="d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb" Oct 31 13:30:20.961604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.961573 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2"] Oct 31 13:30:20.999258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.999214 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-kube-api-access-t7v2s\" (UniqueName: \"kubernetes.io/projected/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-kube-api-access-t7v2s\") pod \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " Oct 31 13:30:20.999393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.999372 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-maas-api-service-ca\") pod \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\" (UID: \"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d\") " Oct 31 13:30:20.999607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.999586 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-kube-api-access-z7hzb\" (UniqueName: \"kubernetes.io/projected/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-kube-api-access-z7hzb\") pod \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " Oct 31 13:30:20.999764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.999744 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-maas-api-service-ca\") pod \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\" (UID: \"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5\") " Oct 31 13:30:21.000020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:20.999990 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-maas-api-service-ca" pod "3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" (UID: "3c6e3708-5e4a-49f1-8849-cb4bd983bc1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:21.000094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.000040 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpnvj\" (UniqueName: \"kubernetes.io/projected/36d64474-1d9d-49ef-84d4-487768f1ab44-kube-api-access-lpnvj\") pod \"maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:21.000094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.000055 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-maas-api-service-ca" pod "6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" (UID: "6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:21.000299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.000279 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d64474-1d9d-49ef-84d4-487768f1ab44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:21.000420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.000335 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2nv\" (UniqueName: \"kubernetes.io/projected/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-kube-api-access-gx2nv\") pod \"maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:21.000479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.000420 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:21.000900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.000820 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.000900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.000846 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.001790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.001768 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:21.002787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.002752 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-kube-api-access-t7v2s" pod "3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" (UID: "3c6e3708-5e4a-49f1-8849-cb4bd983bc1d"). InnerVolumeSpecName "kube-api-access-t7v2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:21.007166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.007140 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-kube-api-access-z7hzb" pod "6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" (UID: "6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5"). InnerVolumeSpecName "kube-api-access-z7hzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:21.013969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.013948 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2nv\" (UniqueName: \"kubernetes.io/projected/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-kube-api-access-gx2nv\") pod \"maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:21.018699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.018368 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:21.087538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.087503 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc"] Oct 31 13:30:21.110318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.109613 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d64474-1d9d-49ef-84d4-487768f1ab44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:21.110318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.109930 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpnvj\" (UniqueName: \"kubernetes.io/projected/36d64474-1d9d-49ef-84d4-487768f1ab44-kube-api-access-lpnvj\") pod \"maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:21.110318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.110002 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hzb\" (UniqueName: \"kubernetes.io/projected/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5-kube-api-access-z7hzb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.110318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.110022 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7v2s\" (UniqueName: \"kubernetes.io/projected/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d-kube-api-access-t7v2s\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.110884 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.110856 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d64474-1d9d-49ef-84d4-487768f1ab44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:21.125118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.125095 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpnvj\" (UniqueName: \"kubernetes.io/projected/36d64474-1d9d-49ef-84d4-487768f1ab44-kube-api-access-lpnvj\") pod \"maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:21.146745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.146709 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerStarted","Data":"1e115098b614268c1362d408147bcfa32d83bbed494b7668a8643fa7eb25c661"} Oct 31 13:30:21.148554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.148533 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" Oct 31 13:30:21.148723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.148686 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs" event={"ID":"3c6e3708-5e4a-49f1-8849-cb4bd983bc1d","Type":"ContainerDied","Data":"cf0017a8e349a00575f6f1c983fef83bb0e763e47418826171e5a90d1225fc33"} Oct 31 13:30:21.148813 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.148752 5074 scope.go:122] "RemoveContainer" containerID="8418e476cf1defbc0e185f9674d9f2422be9185cc444a9c17403790b410b169e" Oct 31 13:30:21.154331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.154311 5074 scope.go:122] "RemoveContainer" containerID="a49f1583ee44512e729208d64807433eec87f0c3941c7c8104399fa397e70f92" Oct 31 13:30:21.154855 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:21.154830 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb_odh-ai-gateway-infra(22fbf28a-3926-4559-aea4-b9e6b5c5c2e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" Oct 31 13:30:21.170550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.170533 5074 scope.go:122] "RemoveContainer" containerID="1bb368f973b63dbf107d7da35f75c8183185e91a997b3e895e75154985642ae2" Oct 31 13:30:21.171327 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:21.171304 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg_odh-ai-gateway-infra(396c4960-19d8-499d-95ce-3e5f995f07fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" Oct 31 13:30:21.175921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.175897 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerStarted","Data":"b2d0b972c7a63314612892c4edf726180c37eae070f0a52ef46f8c387211679a"} Oct 31 13:30:21.178900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.178774 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerStarted","Data":"ca47bd42392b3426a895458b8a1bb99100f7381d408f796e8d65fdc586c5ce1c"} Oct 31 13:30:21.190547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.190514 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" event={"ID":"8b118f65-c0a9-4a84-9f75-f76948b41c8c","Type":"ContainerDied","Data":"f5d61f00cab9aa2a4563eba3c6fa8e0cd133d821352bb4c735b9d6ce9f627986"} Oct 31 13:30:21.190686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.190577 5074 scope.go:122] "RemoveContainer" containerID="d24df5a9114233c8544b6ba24f65979841e51b784cdd2f28c33c13f97f56f9d2" Oct 31 13:30:21.190776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.190751 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd" Oct 31 13:30:21.196318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.196062 5074 scope.go:122] "RemoveContainer" containerID="e4f1836c436edff037c7e023bc68381c7ad6a289df45f0bb6333e8210be1d968" Oct 31 13:30:21.197038 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:21.196610 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9_odh-ai-gateway-infra(6e89d8ad-db41-4694-8db6-524a9c06496f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" Oct 31 13:30:21.204724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.202028 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" event={"ID":"6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5","Type":"ContainerDied","Data":"521bb7d0d7fff5a057565d7e62960757949babe9812072a7b865aa76418a86bc"} Oct 31 13:30:21.204724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.202132 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f" Oct 31 13:30:21.227542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.227511 5074 scope.go:122] "RemoveContainer" containerID="b8ce9c4d2850ec3f536f07e3e162ac8a49c50130c23ddb16737e394c30de49f2" Oct 31 13:30:21.228106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.228083 5074 scope.go:122] "RemoveContainer" containerID="383a2c6556482cb3d5af886de527abc853e81aa62aaeae61975e3d6f6c0ebcef" Oct 31 13:30:21.228518 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:21.228488 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg_odh-ai-gateway-infra(9a2c78e2-3449-4bef-86cd-2d09c24af2ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" Oct 31 13:30:21.238979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.238948 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerStarted","Data":"82d87c1339aafa135aeea86bbfeb3802c0ae9f4679df0bc3a9bea9c5f6e08c30"} Oct 31 13:30:21.286733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.286276 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:21.288798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.288235 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs"] Oct 31 13:30:21.296022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.294649 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890890-8sgxs"] Oct 31 13:30:21.332304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.332224 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq"] Oct 31 13:30:21.352497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.349751 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f"] Oct 31 13:30:21.356876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.355463 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3dd1e5-29890890-jlf6f"] Oct 31 13:30:21.429791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.429221 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7"] Oct 31 13:30:21.442260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.442234 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd"] Oct 31 13:30:21.460779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.460531 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:21.521930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.521891 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-maas-api-service-ca\") pod \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " Oct 31 13:30:21.522135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.522024 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-kube-api-access-sg6pg\" (UniqueName: \"kubernetes.io/projected/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-kube-api-access-sg6pg\") pod \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\" (UID: \"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97\") " Oct 31 13:30:21.522651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.522602 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-maas-api-service-ca" pod "a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" (UID: "a83865ad-ffe7-47f5-8a1d-fb923e0a6b97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:21.524184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.524144 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.540428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.540382 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-569075-29890890-fhkkd"] Oct 31 13:30:21.570991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.570943 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-kube-api-access-sg6pg" pod "a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" (UID: "a83865ad-ffe7-47f5-8a1d-fb923e0a6b97"). InnerVolumeSpecName "kube-api-access-sg6pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:21.628261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.627886 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg6pg\" (UniqueName: \"kubernetes.io/projected/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97-kube-api-access-sg6pg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.644889 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:21.644847 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d64474_1d9d_49ef_84d4_487768f1ab44.slice/crio-ea100a665ca1d5f5f5988263f7e0170038994e9e8f6a52816aecff8667ddea5e WatchSource:0}: Error finding container ea100a665ca1d5f5f5988263f7e0170038994e9e8f6a52816aecff8667ddea5e: Status 404 returned error can't find the container with id ea100a665ca1d5f5f5988263f7e0170038994e9e8f6a52816aecff8667ddea5e Oct 31 13:30:21.647923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.647683 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb"] Oct 31 13:30:21.766455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.766427 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:21.834351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.834238 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646e2652-dde7-413e-97fd-c2ee731dbe67-kube-api-access-28htc\" (UniqueName: \"kubernetes.io/projected/646e2652-dde7-413e-97fd-c2ee731dbe67-kube-api-access-28htc\") pod \"646e2652-dde7-413e-97fd-c2ee731dbe67\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " Oct 31 13:30:21.836749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.836457 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646e2652-dde7-413e-97fd-c2ee731dbe67-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e2652-dde7-413e-97fd-c2ee731dbe67-maas-api-service-ca\") pod \"646e2652-dde7-413e-97fd-c2ee731dbe67\" (UID: \"646e2652-dde7-413e-97fd-c2ee731dbe67\") " Oct 31 13:30:21.838464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.837452 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646e2652-dde7-413e-97fd-c2ee731dbe67-maas-api-service-ca" pod "646e2652-dde7-413e-97fd-c2ee731dbe67" (UID: "646e2652-dde7-413e-97fd-c2ee731dbe67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:21.885993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.885864 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646e2652-dde7-413e-97fd-c2ee731dbe67-kube-api-access-28htc" pod "646e2652-dde7-413e-97fd-c2ee731dbe67" (UID: "646e2652-dde7-413e-97fd-c2ee731dbe67"). InnerVolumeSpecName "kube-api-access-28htc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:21.939693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.939659 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/646e2652-dde7-413e-97fd-c2ee731dbe67-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.940279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.940257 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28htc\" (UniqueName: \"kubernetes.io/projected/646e2652-dde7-413e-97fd-c2ee731dbe67-kube-api-access-28htc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:21.951581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.951545 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4214bb-d6ed-44f0-bcdd-56c754d88f73" path="/var/lib/kubelet/pods/3a4214bb-d6ed-44f0-bcdd-56c754d88f73/volumes" Oct 31 13:30:21.952488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.952461 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" path="/var/lib/kubelet/pods/3c6e3708-5e4a-49f1-8849-cb4bd983bc1d/volumes" Oct 31 13:30:21.953422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.953384 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" path="/var/lib/kubelet/pods/6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5/volumes" Oct 31 13:30:21.954862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.954843 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b118f65-c0a9-4a84-9f75-f76948b41c8c" path="/var/lib/kubelet/pods/8b118f65-c0a9-4a84-9f75-f76948b41c8c/volumes" Oct 31 13:30:21.955617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:21.955594 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2773a8c-6beb-4fe6-844e-200aebb724a3" path="/var/lib/kubelet/pods/e2773a8c-6beb-4fe6-844e-200aebb724a3/volumes" Oct 31 13:30:22.251780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.251730 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerStarted","Data":"d33640e60daaf20426710a40d26847de4b7078616819042539aee25cd80674ee"} Oct 31 13:30:22.254797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.254761 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerStarted","Data":"61b92a6d7cb41782553248d225574a1dd31b9d5a4c2f3a46c8b81c857c6afc71"} Oct 31 13:30:22.257909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.257879 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" event={"ID":"646e2652-dde7-413e-97fd-c2ee731dbe67","Type":"ContainerDied","Data":"b56a59f72ee3094d50f48d1d40ee2d73d928211f63ec054d6bf68fdeed4e63dc"} Oct 31 13:30:22.258047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.257957 5074 scope.go:122] "RemoveContainer" containerID="6fc6907cfdfdb5546947c57702184ce4d170bb14b66e1bfb0b73163281aa1224" Oct 31 13:30:22.258047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.257888 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7" Oct 31 13:30:22.260130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.260099 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" event={"ID":"a83865ad-ffe7-47f5-8a1d-fb923e0a6b97","Type":"ContainerDied","Data":"f3a040fbd80c09d79da2621e422d8a8f9f5f410a175e2f29b6bdf1fa4e2990d4"} Oct 31 13:30:22.260243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.260095 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc" Oct 31 13:30:22.263283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.263252 5074 generic.go:356] "Generic (PLEG): container finished" podID="c7357272-2464-4f33-b723-597f306797bd" containerID="b2d0b972c7a63314612892c4edf726180c37eae070f0a52ef46f8c387211679a" exitCode=60 Oct 31 13:30:22.263443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.263365 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerDied","Data":"b2d0b972c7a63314612892c4edf726180c37eae070f0a52ef46f8c387211679a"} Oct 31 13:30:22.263833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.263815 5074 scope.go:122] "RemoveContainer" containerID="b2d0b972c7a63314612892c4edf726180c37eae070f0a52ef46f8c387211679a" Oct 31 13:30:22.264100 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:22.264080 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r_odh-ai-gateway-infra(c7357272-2464-4f33-b723-597f306797bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" podUID="c7357272-2464-4f33-b723-597f306797bd" Oct 31 13:30:22.268438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.268409 5074 generic.go:356] "Generic (PLEG): container finished" podID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerID="ca47bd42392b3426a895458b8a1bb99100f7381d408f796e8d65fdc586c5ce1c" exitCode=60 Oct 31 13:30:22.268545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.268510 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerDied","Data":"ca47bd42392b3426a895458b8a1bb99100f7381d408f796e8d65fdc586c5ce1c"} Oct 31 13:30:22.268965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.268941 5074 scope.go:122] "RemoveContainer" containerID="ca47bd42392b3426a895458b8a1bb99100f7381d408f796e8d65fdc586c5ce1c" Oct 31 13:30:22.269276 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:22.269251 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm_odh-ai-gateway-infra(ffa79e5f-4f76-4cbb-be95-50a999cb3ccb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" Oct 31 13:30:22.278413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.278382 5074 scope.go:122] "RemoveContainer" containerID="a5f7ff532be40643c30494a2b2ee8b72e8035f4cbe3270008926443ecb20f38c" Oct 31 13:30:22.283224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.283053 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r"] Oct 31 13:30:22.290365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.290312 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" podStartSLOduration=22.290294624 podStartE2EDuration="22.290294624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:22.275248181 +0000 UTC m=+3516.452966168" watchObservedRunningTime="2026-10-31 13:30:22.290294624 +0000 UTC m=+3516.468012612" Oct 31 13:30:22.293047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.292988 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerStarted","Data":"4df5b37bad4ffbc075d551a0f140d58bcd23d90409a1e2c68e5793d019fb1081"} Oct 31 13:30:22.294615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.294586 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" event={"ID":"36d64474-1d9d-49ef-84d4-487768f1ab44","Type":"ContainerStarted","Data":"ea100a665ca1d5f5f5988263f7e0170038994e9e8f6a52816aecff8667ddea5e"} Oct 31 13:30:22.298249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.298224 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerStarted","Data":"0ac8f6b19d9fe3eb25114f2ce98c0bf71c39237cb6d7f74597c4f13318b263c9"} Oct 31 13:30:22.298346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.298268 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerStarted","Data":"b8d06e1af094e9817830f1239a3ede911a55bebe9f1449a9244f04a6363e042d"} Oct 31 13:30:22.323725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.323696 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc"] Oct 31 13:30:22.329343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.329314 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4574ec-29890890-47mjc"] Oct 31 13:30:22.347384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.347288 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" podStartSLOduration=22.347276178 podStartE2EDuration="22.347276178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:22.342807579 +0000 UTC m=+3516.520525579" watchObservedRunningTime="2026-10-31 13:30:22.347276178 +0000 UTC m=+3516.524994162" Oct 31 13:30:22.363345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.363315 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7"] Oct 31 13:30:22.370721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.370689 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-622e03-29890890-p55l7"] Oct 31 13:30:22.378847 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.378820 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm"] Oct 31 13:30:22.429528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.429459 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" podStartSLOduration=22.429446518 podStartE2EDuration="22.429446518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:22.4281393 +0000 UTC m=+3516.605857283" watchObservedRunningTime="2026-10-31 13:30:22.429446518 +0000 UTC m=+3516.607164503" Oct 31 13:30:22.445219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.445187 5074 scope.go:122] "RemoveContainer" containerID="7840ac59912424fac68fde3f52b5e764a459752d5e952e72a730c1bc83de9df6" Oct 31 13:30:22.461141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:22.461116 5074 scope.go:122] "RemoveContainer" containerID="cef952a622b7809d3e6ad7fcabb7a60d8f7b4a7fd76a16482a44ce3afa7b829d" Oct 31 13:30:23.315770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.315738 5074 generic.go:356] "Generic (PLEG): container finished" podID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerID="4df5b37bad4ffbc075d551a0f140d58bcd23d90409a1e2c68e5793d019fb1081" exitCode=60 Oct 31 13:30:23.316180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.315818 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerDied","Data":"4df5b37bad4ffbc075d551a0f140d58bcd23d90409a1e2c68e5793d019fb1081"} Oct 31 13:30:23.316180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.315924 5074 scope.go:122] "RemoveContainer" containerID="d6aac017f2545f79e976ac65d7d2fa030b42e3c52090330e437cb0951d7220bb" Oct 31 13:30:23.316382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.316364 5074 scope.go:122] "RemoveContainer" containerID="4df5b37bad4ffbc075d551a0f140d58bcd23d90409a1e2c68e5793d019fb1081" Oct 31 13:30:23.316803 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:23.316777 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd_odh-ai-gateway-infra(17fa64c7-5eed-46d0-9771-8e5554ee2ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" Oct 31 13:30:23.317778 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.317751 5074 generic.go:356] "Generic (PLEG): container finished" podID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerID="2f1a728dab85dff36211ecd9e8d55629717faac47e7be0f2116f2bc7ddae2f08" exitCode=60 Oct 31 13:30:23.317912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.317829 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" event={"ID":"36d64474-1d9d-49ef-84d4-487768f1ab44","Type":"ContainerDied","Data":"2f1a728dab85dff36211ecd9e8d55629717faac47e7be0f2116f2bc7ddae2f08"} Oct 31 13:30:23.318157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.318135 5074 scope.go:122] "RemoveContainer" containerID="2f1a728dab85dff36211ecd9e8d55629717faac47e7be0f2116f2bc7ddae2f08" Oct 31 13:30:23.320264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.320242 5074 generic.go:356] "Generic (PLEG): container finished" podID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerID="0ac8f6b19d9fe3eb25114f2ce98c0bf71c39237cb6d7f74597c4f13318b263c9" exitCode=60 Oct 31 13:30:23.320419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.320386 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerDied","Data":"0ac8f6b19d9fe3eb25114f2ce98c0bf71c39237cb6d7f74597c4f13318b263c9"} Oct 31 13:30:23.320543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.320525 5074 scope.go:122] "RemoveContainer" containerID="0ac8f6b19d9fe3eb25114f2ce98c0bf71c39237cb6d7f74597c4f13318b263c9" Oct 31 13:30:23.322258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.322237 5074 generic.go:356] "Generic (PLEG): container finished" podID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerID="d33640e60daaf20426710a40d26847de4b7078616819042539aee25cd80674ee" exitCode=60 Oct 31 13:30:23.322371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.322336 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerDied","Data":"d33640e60daaf20426710a40d26847de4b7078616819042539aee25cd80674ee"} Oct 31 13:30:23.322561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.322545 5074 scope.go:122] "RemoveContainer" containerID="d33640e60daaf20426710a40d26847de4b7078616819042539aee25cd80674ee" Oct 31 13:30:23.327088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.326283 5074 generic.go:356] "Generic (PLEG): container finished" podID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerID="61b92a6d7cb41782553248d225574a1dd31b9d5a4c2f3a46c8b81c857c6afc71" exitCode=60 Oct 31 13:30:23.327088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.326397 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerDied","Data":"61b92a6d7cb41782553248d225574a1dd31b9d5a4c2f3a46c8b81c857c6afc71"} Oct 31 13:30:23.327088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.326626 5074 scope.go:122] "RemoveContainer" containerID="61b92a6d7cb41782553248d225574a1dd31b9d5a4c2f3a46c8b81c857c6afc71" Oct 31 13:30:23.446300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.446265 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd"] Oct 31 13:30:23.595048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.595016 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:23.645619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.645593 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:23.680868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.680832 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7357272-2464-4f33-b723-597f306797bd-kube-api-access-8mt8f\" (UniqueName: \"kubernetes.io/projected/c7357272-2464-4f33-b723-597f306797bd-kube-api-access-8mt8f\") pod \"c7357272-2464-4f33-b723-597f306797bd\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " Oct 31 13:30:23.681350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.681277 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7357272-2464-4f33-b723-597f306797bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7357272-2464-4f33-b723-597f306797bd-maas-api-service-ca\") pod \"c7357272-2464-4f33-b723-597f306797bd\" (UID: \"c7357272-2464-4f33-b723-597f306797bd\") " Oct 31 13:30:23.681706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.681683 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-kube-api-access-crvjz\" (UniqueName: \"kubernetes.io/projected/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-kube-api-access-crvjz\") pod \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " Oct 31 13:30:23.682038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.682007 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-maas-api-service-ca\") pod \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\" (UID: \"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb\") " Oct 31 13:30:23.683958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.683934 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7357272-2464-4f33-b723-597f306797bd-maas-api-service-ca" pod "c7357272-2464-4f33-b723-597f306797bd" (UID: "c7357272-2464-4f33-b723-597f306797bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:23.685343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.685320 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-maas-api-service-ca" pod "ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" (UID: "ffa79e5f-4f76-4cbb-be95-50a999cb3ccb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:23.785563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.785528 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7357272-2464-4f33-b723-597f306797bd-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.785563 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.785561 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.893823 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.893111 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-kube-api-access-crvjz" pod "ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" (UID: "ffa79e5f-4f76-4cbb-be95-50a999cb3ccb"). InnerVolumeSpecName "kube-api-access-crvjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:23.899047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.897601 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvjz\" (UniqueName: \"kubernetes.io/projected/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb-kube-api-access-crvjz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.901947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.901904 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7357272-2464-4f33-b723-597f306797bd-kube-api-access-8mt8f" pod "c7357272-2464-4f33-b723-597f306797bd" (UID: "c7357272-2464-4f33-b723-597f306797bd"). InnerVolumeSpecName "kube-api-access-8mt8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:23.946728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.946692 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" path="/var/lib/kubelet/pods/646e2652-dde7-413e-97fd-c2ee731dbe67/volumes" Oct 31 13:30:23.947448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:23.947420 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" path="/var/lib/kubelet/pods/a83865ad-ffe7-47f5-8a1d-fb923e0a6b97/volumes" Oct 31 13:30:24.000265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.000237 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mt8f\" (UniqueName: \"kubernetes.io/projected/c7357272-2464-4f33-b723-597f306797bd-kube-api-access-8mt8f\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:24.341493 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.341456 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" Oct 31 13:30:24.341493 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.341483 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r" event={"ID":"c7357272-2464-4f33-b723-597f306797bd","Type":"ContainerDied","Data":"8632e4befc3ff22561819847700353748d2d4130740c7b285e23142f5b33f54c"} Oct 31 13:30:24.342050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.341555 5074 scope.go:122] "RemoveContainer" containerID="b2d0b972c7a63314612892c4edf726180c37eae070f0a52ef46f8c387211679a" Oct 31 13:30:24.343576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.343553 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" Oct 31 13:30:24.344043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.343627 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm" event={"ID":"ffa79e5f-4f76-4cbb-be95-50a999cb3ccb","Type":"ContainerDied","Data":"6df3eab750996056980686d4483f93b65280b0ed903dcbaf7d526cfe7c116614"} Oct 31 13:30:24.348725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.348695 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" event={"ID":"36d64474-1d9d-49ef-84d4-487768f1ab44","Type":"ContainerStarted","Data":"b29fb08b7359e8d01e4ab3635e46756d109294dbd554da55662c69f6416bfa11"} Oct 31 13:30:24.351082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.350951 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerStarted","Data":"f8c8e47bc300235c9677a1c43ddb1baf9d10e93bf808e667cff2c4ed3ea16186"} Oct 31 13:30:24.353166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.353141 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerStarted","Data":"a508c190f8e4b5e7875e2d93aaf6ee974ef7164a1b77165fba7ac2bed9f24f08"} Oct 31 13:30:24.355257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.355234 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerStarted","Data":"fa67278db9e2043c451c46e2048c8dfb2ce2a03226fce52163f390de7f1abfd5"} Oct 31 13:30:24.357643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.357609 5074 scope.go:122] "RemoveContainer" containerID="ca47bd42392b3426a895458b8a1bb99100f7381d408f796e8d65fdc586c5ce1c" Oct 31 13:30:24.401154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.401119 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r"] Oct 31 13:30:24.409649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.408787 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2ef686-29890890-tnr6r"] Oct 31 13:30:24.433923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.433856 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" podStartSLOduration=24.433836544 podStartE2EDuration="24.433836544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:24.431206147 +0000 UTC m=+3518.608924158" watchObservedRunningTime="2026-10-31 13:30:24.433836544 +0000 UTC m=+3518.611554529" Oct 31 13:30:24.513147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.513114 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm"] Oct 31 13:30:24.519046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.519019 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5452fc-29890890-zkbvm"] Oct 31 13:30:24.805360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.805333 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:24.920134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.920048 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-kube-api-access-x7pnw\" (UniqueName: \"kubernetes.io/projected/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-kube-api-access-x7pnw\") pod \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " Oct 31 13:30:24.920301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.920250 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-maas-api-service-ca\") pod \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\" (UID: \"17fa64c7-5eed-46d0-9771-8e5554ee2ac8\") " Oct 31 13:30:24.920738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.920700 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-maas-api-service-ca" pod "17fa64c7-5eed-46d0-9771-8e5554ee2ac8" (UID: "17fa64c7-5eed-46d0-9771-8e5554ee2ac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:24.921235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.921210 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:24.922403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:24.922371 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-kube-api-access-x7pnw" pod "17fa64c7-5eed-46d0-9771-8e5554ee2ac8" (UID: "17fa64c7-5eed-46d0-9771-8e5554ee2ac8"). InnerVolumeSpecName "kube-api-access-x7pnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:25.023270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.023232 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7pnw\" (UniqueName: \"kubernetes.io/projected/17fa64c7-5eed-46d0-9771-8e5554ee2ac8-kube-api-access-x7pnw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:25.367958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.367917 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" event={"ID":"17fa64c7-5eed-46d0-9771-8e5554ee2ac8","Type":"ContainerDied","Data":"94e8674259aa16ed79b01fbeff7fa6120d2c4f8b832da0bee12d1094c21c3aa7"} Oct 31 13:30:25.368576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.367984 5074 scope.go:122] "RemoveContainer" containerID="4df5b37bad4ffbc075d551a0f140d58bcd23d90409a1e2c68e5793d019fb1081" Oct 31 13:30:25.368576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.367933 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd" Oct 31 13:30:25.370321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.370294 5074 generic.go:356] "Generic (PLEG): container finished" podID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerID="b29fb08b7359e8d01e4ab3635e46756d109294dbd554da55662c69f6416bfa11" exitCode=60 Oct 31 13:30:25.370458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.370387 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" event={"ID":"36d64474-1d9d-49ef-84d4-487768f1ab44","Type":"ContainerDied","Data":"b29fb08b7359e8d01e4ab3635e46756d109294dbd554da55662c69f6416bfa11"} Oct 31 13:30:25.370782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.370764 5074 scope.go:122] "RemoveContainer" containerID="b29fb08b7359e8d01e4ab3635e46756d109294dbd554da55662c69f6416bfa11" Oct 31 13:30:25.371116 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:25.371087 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb_odh-ai-gateway-infra(36d64474-1d9d-49ef-84d4-487768f1ab44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" Oct 31 13:30:25.372977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.372949 5074 generic.go:356] "Generic (PLEG): container finished" podID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerID="f8c8e47bc300235c9677a1c43ddb1baf9d10e93bf808e667cff2c4ed3ea16186" exitCode=60 Oct 31 13:30:25.373118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.373099 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerDied","Data":"f8c8e47bc300235c9677a1c43ddb1baf9d10e93bf808e667cff2c4ed3ea16186"} Oct 31 13:30:25.373379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.373355 5074 scope.go:122] "RemoveContainer" containerID="f8c8e47bc300235c9677a1c43ddb1baf9d10e93bf808e667cff2c4ed3ea16186" Oct 31 13:30:25.373877 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:25.373689 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq_odh-ai-gateway-infra(aa75011f-6c07-44fd-b5e7-7b2edda5e2e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" Oct 31 13:30:25.375448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.375428 5074 generic.go:356] "Generic (PLEG): container finished" podID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerID="a508c190f8e4b5e7875e2d93aaf6ee974ef7164a1b77165fba7ac2bed9f24f08" exitCode=60 Oct 31 13:30:25.375554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.375502 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerDied","Data":"a508c190f8e4b5e7875e2d93aaf6ee974ef7164a1b77165fba7ac2bed9f24f08"} Oct 31 13:30:25.375889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.375872 5074 scope.go:122] "RemoveContainer" containerID="a508c190f8e4b5e7875e2d93aaf6ee974ef7164a1b77165fba7ac2bed9f24f08" Oct 31 13:30:25.376134 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:25.376114 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2_odh-ai-gateway-infra(5a23e0a1-17ab-434b-aa91-1ce660969ec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" Oct 31 13:30:25.378243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.378217 5074 generic.go:356] "Generic (PLEG): container finished" podID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerID="fa67278db9e2043c451c46e2048c8dfb2ce2a03226fce52163f390de7f1abfd5" exitCode=60 Oct 31 13:30:25.378356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.378302 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerDied","Data":"fa67278db9e2043c451c46e2048c8dfb2ce2a03226fce52163f390de7f1abfd5"} Oct 31 13:30:25.378613 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.378593 5074 scope.go:122] "RemoveContainer" containerID="fa67278db9e2043c451c46e2048c8dfb2ce2a03226fce52163f390de7f1abfd5" Oct 31 13:30:25.378968 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:25.378946 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg_odh-ai-gateway-infra(3d7495d8-6556-4de7-aac1-d560a8b63fb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" Oct 31 13:30:25.387208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.387178 5074 scope.go:122] "RemoveContainer" containerID="2f1a728dab85dff36211ecd9e8d55629717faac47e7be0f2116f2bc7ddae2f08" Oct 31 13:30:25.403739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.403706 5074 scope.go:122] "RemoveContainer" containerID="0ac8f6b19d9fe3eb25114f2ce98c0bf71c39237cb6d7f74597c4f13318b263c9" Oct 31 13:30:25.423124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.423104 5074 scope.go:122] "RemoveContainer" containerID="d33640e60daaf20426710a40d26847de4b7078616819042539aee25cd80674ee" Oct 31 13:30:25.438910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.438857 5074 scope.go:122] "RemoveContainer" containerID="61b92a6d7cb41782553248d225574a1dd31b9d5a4c2f3a46c8b81c857c6afc71" Oct 31 13:30:25.488419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.488381 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd"] Oct 31 13:30:25.496410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.496373 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-7cdec6-29890890-dv6fd"] Oct 31 13:30:25.944079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.944045 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" path="/var/lib/kubelet/pods/17fa64c7-5eed-46d0-9771-8e5554ee2ac8/volumes" Oct 31 13:30:25.944576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.944558 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7357272-2464-4f33-b723-597f306797bd" path="/var/lib/kubelet/pods/c7357272-2464-4f33-b723-597f306797bd/volumes" Oct 31 13:30:25.945089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:25.945066 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" path="/var/lib/kubelet/pods/ffa79e5f-4f76-4cbb-be95-50a999cb3ccb/volumes" Oct 31 13:30:25.985195 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:25.985155 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:30:26.393691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:26.393657 5074 scope.go:122] "RemoveContainer" containerID="f8c8e47bc300235c9677a1c43ddb1baf9d10e93bf808e667cff2c4ed3ea16186" Oct 31 13:30:26.394177 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:26.394133 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq_odh-ai-gateway-infra(aa75011f-6c07-44fd-b5e7-7b2edda5e2e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" Oct 31 13:30:26.395814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:26.395788 5074 scope.go:122] "RemoveContainer" containerID="a508c190f8e4b5e7875e2d93aaf6ee974ef7164a1b77165fba7ac2bed9f24f08" Oct 31 13:30:26.396197 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:26.396172 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2_odh-ai-gateway-infra(5a23e0a1-17ab-434b-aa91-1ce660969ec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" Oct 31 13:30:26.397903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:26.397879 5074 scope.go:122] "RemoveContainer" containerID="fa67278db9e2043c451c46e2048c8dfb2ce2a03226fce52163f390de7f1abfd5" Oct 31 13:30:26.398317 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:26.398289 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg_odh-ai-gateway-infra(3d7495d8-6556-4de7-aac1-d560a8b63fb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" Oct 31 13:30:26.401333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:26.401313 5074 scope.go:122] "RemoveContainer" containerID="b29fb08b7359e8d01e4ab3635e46756d109294dbd554da55662c69f6416bfa11" Oct 31 13:30:26.401695 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:26.401619 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb_odh-ai-gateway-infra(36d64474-1d9d-49ef-84d4-487768f1ab44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" Oct 31 13:30:28.027356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:28.027317 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p"] Oct 31 13:30:28.032835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:28.032803 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d9a7a5-m544p"] Oct 31 13:30:29.942252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:29.942216 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91527f14-f515-4b3e-aa72-55bc824113a9" path="/var/lib/kubelet/pods/91527f14-f515-4b3e-aa72-55bc824113a9/volumes" Oct 31 13:30:29.943233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:29.943215 5074 scope.go:122] "RemoveContainer" containerID="e4f1836c436edff037c7e023bc68381c7ad6a289df45f0bb6333e8210be1d968" Oct 31 13:30:30.507344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:30.507308 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" event={"ID":"6e89d8ad-db41-4694-8db6-524a9c06496f","Type":"ContainerStarted","Data":"b02e4d1ea765b4ef3c852be8722fb680d909320d123904126a166b99e6fa1347"} Oct 31 13:30:31.524962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:31.524924 5074 generic.go:356] "Generic (PLEG): container finished" podID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerID="b02e4d1ea765b4ef3c852be8722fb680d909320d123904126a166b99e6fa1347" exitCode=60 Oct 31 13:30:31.525445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:31.525007 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" event={"ID":"6e89d8ad-db41-4694-8db6-524a9c06496f","Type":"ContainerDied","Data":"b02e4d1ea765b4ef3c852be8722fb680d909320d123904126a166b99e6fa1347"} Oct 31 13:30:31.525445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:31.525070 5074 scope.go:122] "RemoveContainer" containerID="e4f1836c436edff037c7e023bc68381c7ad6a289df45f0bb6333e8210be1d968" Oct 31 13:30:31.525601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:31.525582 5074 scope.go:122] "RemoveContainer" containerID="b02e4d1ea765b4ef3c852be8722fb680d909320d123904126a166b99e6fa1347" Oct 31 13:30:31.526337 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:31.526079 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9_odh-ai-gateway-infra(6e89d8ad-db41-4694-8db6-524a9c06496f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" Oct 31 13:30:31.567594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:31.567556 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9"] Oct 31 13:30:31.942537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:31.942460 5074 scope.go:122] "RemoveContainer" containerID="a49f1583ee44512e729208d64807433eec87f0c3941c7c8104399fa397e70f92" Oct 31 13:30:32.538731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.538627 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" event={"ID":"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3","Type":"ContainerStarted","Data":"ae50c0dfad130771f1a57c2ffda42746450c72bbf7bc7b45337bd62a781cd7a9"} Oct 31 13:30:32.626921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.626852 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" podStartSLOduration=32.626833396 podStartE2EDuration="32.626833396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:32.619451314 +0000 UTC m=+3526.797169298" watchObservedRunningTime="2026-10-31 13:30:32.626833396 +0000 UTC m=+3526.804551381" Oct 31 13:30:32.768452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.768424 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:32.841513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.841481 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e89d8ad-db41-4694-8db6-524a9c06496f-kube-api-access-r9xmm\" (UniqueName: \"kubernetes.io/projected/6e89d8ad-db41-4694-8db6-524a9c06496f-kube-api-access-r9xmm\") pod \"6e89d8ad-db41-4694-8db6-524a9c06496f\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " Oct 31 13:30:32.841738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.841529 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e89d8ad-db41-4694-8db6-524a9c06496f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e89d8ad-db41-4694-8db6-524a9c06496f-maas-api-service-ca\") pod \"6e89d8ad-db41-4694-8db6-524a9c06496f\" (UID: \"6e89d8ad-db41-4694-8db6-524a9c06496f\") " Oct 31 13:30:32.842083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.842048 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e89d8ad-db41-4694-8db6-524a9c06496f-maas-api-service-ca" pod "6e89d8ad-db41-4694-8db6-524a9c06496f" (UID: "6e89d8ad-db41-4694-8db6-524a9c06496f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:32.842221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.842202 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e89d8ad-db41-4694-8db6-524a9c06496f-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:32.843911 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.843878 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e89d8ad-db41-4694-8db6-524a9c06496f-kube-api-access-r9xmm" pod "6e89d8ad-db41-4694-8db6-524a9c06496f" (UID: "6e89d8ad-db41-4694-8db6-524a9c06496f"). InnerVolumeSpecName "kube-api-access-r9xmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:32.932286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.932245 5074 scope.go:122] "RemoveContainer" containerID="383a2c6556482cb3d5af886de527abc853e81aa62aaeae61975e3d6f6c0ebcef" Oct 31 13:30:32.932579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.932553 5074 scope.go:122] "RemoveContainer" containerID="a9d8ffd97317cc4a4d59282b3798dc668465ecd6c92e3e3d83c5758eb71128f7" Oct 31 13:30:32.932765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.932673 5074 scope.go:122] "RemoveContainer" containerID="1bb368f973b63dbf107d7da35f75c8183185e91a997b3e895e75154985642ae2" Oct 31 13:30:32.943556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:32.943527 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9xmm\" (UniqueName: \"kubernetes.io/projected/6e89d8ad-db41-4694-8db6-524a9c06496f-kube-api-access-r9xmm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:33.548709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.548582 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" Oct 31 13:30:33.548709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.548615 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9" event={"ID":"6e89d8ad-db41-4694-8db6-524a9c06496f","Type":"ContainerDied","Data":"7942cc2c74ff51b15ddc58f27a92ed3dff99daeb7a2681b7785f1a98b3ce7224"} Oct 31 13:30:33.548709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.548697 5074 scope.go:122] "RemoveContainer" containerID="b02e4d1ea765b4ef3c852be8722fb680d909320d123904126a166b99e6fa1347" Oct 31 13:30:33.550969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.550941 5074 generic.go:356] "Generic (PLEG): container finished" podID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerID="ae50c0dfad130771f1a57c2ffda42746450c72bbf7bc7b45337bd62a781cd7a9" exitCode=60 Oct 31 13:30:33.551198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.551179 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" event={"ID":"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3","Type":"ContainerDied","Data":"ae50c0dfad130771f1a57c2ffda42746450c72bbf7bc7b45337bd62a781cd7a9"} Oct 31 13:30:33.552228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.551747 5074 scope.go:122] "RemoveContainer" containerID="ae50c0dfad130771f1a57c2ffda42746450c72bbf7bc7b45337bd62a781cd7a9" Oct 31 13:30:33.552228 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:33.552182 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb_odh-ai-gateway-infra(22fbf28a-3926-4559-aea4-b9e6b5c5c2e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" Oct 31 13:30:33.553361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.553339 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerStarted","Data":"6642acc36c13018ed292ea7efea81ffdcad7378bdad560c987d4a9baf251eff1"} Oct 31 13:30:33.561546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.561530 5074 scope.go:122] "RemoveContainer" containerID="a49f1583ee44512e729208d64807433eec87f0c3941c7c8104399fa397e70f92" Oct 31 13:30:33.608185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.608097 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9"] Oct 31 13:30:33.619643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.619599 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4023ad-29890890-ktcg9"] Oct 31 13:30:33.643038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.643012 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb"] Oct 31 13:30:33.942337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:33.942260 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" path="/var/lib/kubelet/pods/6e89d8ad-db41-4694-8db6-524a9c06496f/volumes" Oct 31 13:30:34.595658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.582756 5074 generic.go:356] "Generic (PLEG): container finished" podID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerID="b1bbb06a1a9f569f9505adc8f4dc6e2c11c585e2f3da6732e3a11457f4c47ec3" exitCode=60 Oct 31 13:30:34.595658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.582870 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" event={"ID":"086f29d4-9e94-45fd-a9cb-d36f19841d44","Type":"ContainerDied","Data":"b1bbb06a1a9f569f9505adc8f4dc6e2c11c585e2f3da6732e3a11457f4c47ec3"} Oct 31 13:30:34.595658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.582927 5074 scope.go:122] "RemoveContainer" containerID="a9d8ffd97317cc4a4d59282b3798dc668465ecd6c92e3e3d83c5758eb71128f7" Oct 31 13:30:34.626654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.623013 5074 generic.go:356] "Generic (PLEG): container finished" podID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerID="5cc6b66c5068d066d953ec3439af5916b6cf79a7903d99ef304f052cd23fe9d4" exitCode=60 Oct 31 13:30:34.626654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.623216 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" event={"ID":"9a2c78e2-3449-4bef-86cd-2d09c24af2ac","Type":"ContainerDied","Data":"5cc6b66c5068d066d953ec3439af5916b6cf79a7903d99ef304f052cd23fe9d4"} Oct 31 13:30:34.647054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.636616 5074 scope.go:122] "RemoveContainer" containerID="b1bbb06a1a9f569f9505adc8f4dc6e2c11c585e2f3da6732e3a11457f4c47ec3" Oct 31 13:30:34.647054 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:34.637463 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw_odh-ai-gateway-infra(086f29d4-9e94-45fd-a9cb-d36f19841d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" Oct 31 13:30:34.647054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.643744 5074 scope.go:122] "RemoveContainer" containerID="5cc6b66c5068d066d953ec3439af5916b6cf79a7903d99ef304f052cd23fe9d4" Oct 31 13:30:34.647054 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:34.644167 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg_odh-ai-gateway-infra(9a2c78e2-3449-4bef-86cd-2d09c24af2ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" Oct 31 13:30:34.673602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.668516 5074 generic.go:356] "Generic (PLEG): container finished" podID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerID="6642acc36c13018ed292ea7efea81ffdcad7378bdad560c987d4a9baf251eff1" exitCode=60 Oct 31 13:30:34.674007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.673940 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerDied","Data":"6642acc36c13018ed292ea7efea81ffdcad7378bdad560c987d4a9baf251eff1"} Oct 31 13:30:34.698590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.698556 5074 scope.go:122] "RemoveContainer" containerID="6642acc36c13018ed292ea7efea81ffdcad7378bdad560c987d4a9baf251eff1" Oct 31 13:30:34.699084 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:34.699045 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg_odh-ai-gateway-infra(396c4960-19d8-499d-95ce-3e5f995f07fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" Oct 31 13:30:34.705982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.705946 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg"] Oct 31 13:30:34.742936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.742800 5074 scope.go:122] "RemoveContainer" containerID="383a2c6556482cb3d5af886de527abc853e81aa62aaeae61975e3d6f6c0ebcef" Oct 31 13:30:34.764797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.764776 5074 scope.go:122] "RemoveContainer" containerID="1bb368f973b63dbf107d7da35f75c8183185e91a997b3e895e75154985642ae2" Oct 31 13:30:34.937668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.937610 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:34.948792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.948766 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-kube-api-access-d5w2d\" (UniqueName: \"kubernetes.io/projected/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-kube-api-access-d5w2d\") pod \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " Oct 31 13:30:34.949014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.948999 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-maas-api-service-ca\") pod \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\" (UID: \"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3\") " Oct 31 13:30:34.949342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.949319 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-maas-api-service-ca" pod "22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" (UID: "22fbf28a-3926-4559-aea4-b9e6b5c5c2e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:34.949579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.949558 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:34.950868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:34.950848 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-kube-api-access-d5w2d" pod "22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" (UID: "22fbf28a-3926-4559-aea4-b9e6b5c5c2e3"). InnerVolumeSpecName "kube-api-access-d5w2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:35.051508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.051431 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5w2d\" (UniqueName: \"kubernetes.io/projected/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3-kube-api-access-d5w2d\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:35.636683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.636647 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw"] Oct 31 13:30:35.691564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.691538 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" Oct 31 13:30:35.691828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.691612 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb" event={"ID":"22fbf28a-3926-4559-aea4-b9e6b5c5c2e3","Type":"ContainerDied","Data":"a8677ee72ddcd98bfa1e296fca9d0a820f363aa7142c35d001bfa38315ddf15f"} Oct 31 13:30:35.691997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.691892 5074 scope.go:122] "RemoveContainer" containerID="ae50c0dfad130771f1a57c2ffda42746450c72bbf7bc7b45337bd62a781cd7a9" Oct 31 13:30:35.789819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.787386 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb"] Oct 31 13:30:35.818777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.818737 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg"] Oct 31 13:30:35.825811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.825735 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5a9e45-29890890-87bqb"] Oct 31 13:30:35.948726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.948690 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" path="/var/lib/kubelet/pods/22fbf28a-3926-4559-aea4-b9e6b5c5c2e3/volumes" Oct 31 13:30:35.951166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.951145 5074 scope.go:122] "RemoveContainer" containerID="a508c190f8e4b5e7875e2d93aaf6ee974ef7164a1b77165fba7ac2bed9f24f08" Oct 31 13:30:35.980681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:35.980381 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:36.078734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.077555 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086f29d4-9e94-45fd-a9cb-d36f19841d44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086f29d4-9e94-45fd-a9cb-d36f19841d44-maas-api-service-ca\") pod \"086f29d4-9e94-45fd-a9cb-d36f19841d44\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " Oct 31 13:30:36.078734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.077813 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086f29d4-9e94-45fd-a9cb-d36f19841d44-kube-api-access-98z9n\" (UniqueName: \"kubernetes.io/projected/086f29d4-9e94-45fd-a9cb-d36f19841d44-kube-api-access-98z9n\") pod \"086f29d4-9e94-45fd-a9cb-d36f19841d44\" (UID: \"086f29d4-9e94-45fd-a9cb-d36f19841d44\") " Oct 31 13:30:36.078734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.077984 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086f29d4-9e94-45fd-a9cb-d36f19841d44-maas-api-service-ca" pod "086f29d4-9e94-45fd-a9cb-d36f19841d44" (UID: "086f29d4-9e94-45fd-a9cb-d36f19841d44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:36.078734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.078704 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086f29d4-9e94-45fd-a9cb-d36f19841d44-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.170821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.170738 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086f29d4-9e94-45fd-a9cb-d36f19841d44-kube-api-access-98z9n" pod "086f29d4-9e94-45fd-a9cb-d36f19841d44" (UID: "086f29d4-9e94-45fd-a9cb-d36f19841d44"). InnerVolumeSpecName "kube-api-access-98z9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:36.180732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.180685 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98z9n\" (UniqueName: \"kubernetes.io/projected/086f29d4-9e94-45fd-a9cb-d36f19841d44-kube-api-access-98z9n\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.216028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.215992 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:36.227220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.227194 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:36.254233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.254202 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7"] Oct 31 13:30:36.255721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255693 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:36.255920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255892 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255935 5074 cpu_manager.go:414] "containerMap: removing container" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255944 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255965 5074 cpu_manager.go:414] "containerMap: removing container" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255972 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255987 5074 cpu_manager.go:414] "containerMap: removing container" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.255994 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256006 5074 cpu_manager.go:414] "containerMap: removing container" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256013 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256028 5074 cpu_manager.go:414] "containerMap: removing container" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256035 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256052 5074 cpu_manager.go:414] "containerMap: removing container" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.256055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256059 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256070 5074 cpu_manager.go:414] "containerMap: removing container" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256077 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256089 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256097 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256108 5074 cpu_manager.go:414] "containerMap: removing container" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256115 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256126 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256133 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256143 5074 cpu_manager.go:414] "containerMap: removing container" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256150 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256169 5074 cpu_manager.go:414] "containerMap: removing container" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256175 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256187 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256193 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256203 5074 cpu_manager.go:414] "containerMap: removing container" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256210 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256221 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256229 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256248 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256295 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256316 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256325 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256340 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256348 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256356 5074 cpu_manager.go:414] "containerMap: removing container" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256364 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256405 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256412 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256425 5074 cpu_manager.go:414] "containerMap: removing container" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256433 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256448 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256455 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256464 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256471 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256486 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:36.256561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256494 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256740 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256757 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256770 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256784 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256793 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256804 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256814 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256823 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256831 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256840 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6e3708-5e4a-49f1-8849-cb4bd983bc1d" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256852 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256861 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc57768-d06e-47a4-b4ff-87c1f5cf4ca5" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256872 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa79e5f-4f76-4cbb-be95-50a999cb3ccb" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256883 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256892 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256903 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256911 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256923 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256936 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256947 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256958 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256968 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256976 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256985 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.256994 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:36.257872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.257005 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fbf28a-3926-4559-aea4-b9e6b5c5c2e3" containerName="cleanup" Oct 31 13:30:36.291755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.291720 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396c4960-19d8-499d-95ce-3e5f995f07fc-kube-api-access-b4974\" (UniqueName: \"kubernetes.io/projected/396c4960-19d8-499d-95ce-3e5f995f07fc-kube-api-access-b4974\") pod \"396c4960-19d8-499d-95ce-3e5f995f07fc\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " Oct 31 13:30:36.295872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.295838 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-maas-api-service-ca\") pod \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " Oct 31 13:30:36.296204 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.296185 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-kube-api-access-f6fs2\" (UniqueName: \"kubernetes.io/projected/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-kube-api-access-f6fs2\") pod \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\" (UID: \"9a2c78e2-3449-4bef-86cd-2d09c24af2ac\") " Oct 31 13:30:36.296393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.296378 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396c4960-19d8-499d-95ce-3e5f995f07fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396c4960-19d8-499d-95ce-3e5f995f07fc-maas-api-service-ca\") pod \"396c4960-19d8-499d-95ce-3e5f995f07fc\" (UID: \"396c4960-19d8-499d-95ce-3e5f995f07fc\") " Oct 31 13:30:36.297696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.297672 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396c4960-19d8-499d-95ce-3e5f995f07fc-maas-api-service-ca" pod "396c4960-19d8-499d-95ce-3e5f995f07fc" (UID: "396c4960-19d8-499d-95ce-3e5f995f07fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:36.298077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.298050 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-maas-api-service-ca" pod "9a2c78e2-3449-4bef-86cd-2d09c24af2ac" (UID: "9a2c78e2-3449-4bef-86cd-2d09c24af2ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:36.303612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.303592 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.305212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.305189 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7"] Oct 31 13:30:36.307572 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.307548 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-kube-api-access-f6fs2" pod "9a2c78e2-3449-4bef-86cd-2d09c24af2ac" (UID: "9a2c78e2-3449-4bef-86cd-2d09c24af2ac"). InnerVolumeSpecName "kube-api-access-f6fs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:36.308109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.308079 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396c4960-19d8-499d-95ce-3e5f995f07fc-kube-api-access-b4974" pod "396c4960-19d8-499d-95ce-3e5f995f07fc" (UID: "396c4960-19d8-499d-95ce-3e5f995f07fc"). InnerVolumeSpecName "kube-api-access-b4974". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:36.318173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.317409 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-09db67-openshift-default-dockercfg-2chg5\"" Oct 31 13:30:36.400212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400162 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-credential-socket\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400243 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-token\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400379 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa965c2-ad6a-4339-879f-41318554f6ce-istiod-ca-cert\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400543 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-podinfo\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400584 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwvp\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-kube-api-access-dbwvp\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400624 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-certs\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400754 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-envoy\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400805 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-socket\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.400978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.400922 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-data\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.401049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.401036 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.401109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.401052 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fs2\" (UniqueName: \"kubernetes.io/projected/9a2c78e2-3449-4bef-86cd-2d09c24af2ac-kube-api-access-f6fs2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.401109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.401063 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/396c4960-19d8-499d-95ce-3e5f995f07fc-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.401109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.401073 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4974\" (UniqueName: \"kubernetes.io/projected/396c4960-19d8-499d-95ce-3e5f995f07fc-kube-api-access-b4974\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:36.512485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512442 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-podinfo\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.512686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512525 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwvp\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-kube-api-access-dbwvp\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.512686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512569 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-certs\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.512686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512645 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-envoy\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.512819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512685 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-socket\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.512819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512780 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-data\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.512893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512852 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-credential-socket\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.512926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.512913 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-token\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.513027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.513009 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa965c2-ad6a-4339-879f-41318554f6ce-istiod-ca-cert\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.527702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.525015 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa965c2-ad6a-4339-879f-41318554f6ce-istiod-ca-cert\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.531653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.528043 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-socket\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.531653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.528306 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-data\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.531653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.528544 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-credential-socket\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.531653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.529067 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-certs\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.541761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.541734 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-token\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.546000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.545971 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-envoy\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.547855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.547266 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-podinfo\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.548072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.547807 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwvp\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-kube-api-access-dbwvp\") pod \"e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.654040 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.653993 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:36.704882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.704708 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" event={"ID":"396c4960-19d8-499d-95ce-3e5f995f07fc","Type":"ContainerDied","Data":"1615dca522560993cc9a849b4c4baad10014fc8d1f96e0a34a5e820daad0940d"} Oct 31 13:30:36.704882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.704880 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg" Oct 31 13:30:36.705659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.704887 5074 scope.go:122] "RemoveContainer" containerID="6642acc36c13018ed292ea7efea81ffdcad7378bdad560c987d4a9baf251eff1" Oct 31 13:30:36.712699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.710851 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" event={"ID":"086f29d4-9e94-45fd-a9cb-d36f19841d44","Type":"ContainerDied","Data":"913715d25f8350588bec14ee95849617ca74ba487f817bf59148405632d6b338"} Oct 31 13:30:36.712699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.710990 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw" Oct 31 13:30:36.728281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.728248 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" event={"ID":"9a2c78e2-3449-4bef-86cd-2d09c24af2ac","Type":"ContainerDied","Data":"c0a268373dd2a558f73e52a5d6799e3f8e31384ff49e148a8d7cacb526994729"} Oct 31 13:30:36.728443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.728301 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg" Oct 31 13:30:36.730478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.730452 5074 scope.go:122] "RemoveContainer" containerID="b1bbb06a1a9f569f9505adc8f4dc6e2c11c585e2f3da6732e3a11457f4c47ec3" Oct 31 13:30:36.732857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.732829 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerStarted","Data":"205be1be5222d3164a067ccf84da242ea6e8af7b96b15cd466b43831af0fe2b9"} Oct 31 13:30:36.749761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.749735 5074 scope.go:122] "RemoveContainer" containerID="5cc6b66c5068d066d953ec3439af5916b6cf79a7903d99ef304f052cd23fe9d4" Oct 31 13:30:36.789004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.788976 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg"] Oct 31 13:30:36.802164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.801431 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c6c445-29890890-9q2jg"] Oct 31 13:30:36.812021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.811985 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg"] Oct 31 13:30:36.825768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.825737 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-691f1e-29890890-gr8lg"] Oct 31 13:30:36.853068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.853040 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw"] Oct 31 13:30:36.858080 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.857706 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-842ac9-29890890-vxpzw"] Oct 31 13:30:36.931233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.931199 5074 scope.go:122] "RemoveContainer" containerID="f8c8e47bc300235c9677a1c43ddb1baf9d10e93bf808e667cff2c4ed3ea16186" Oct 31 13:30:36.960816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.960792 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7"] Oct 31 13:30:36.963084 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:36.963055 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa965c2_ad6a_4339_879f_41318554f6ce.slice/crio-5f77cdd761d9a523adca05c1f0fa95fdba9bc11671864577ff3025d3ebcbe431 WatchSource:0}: Error finding container 5f77cdd761d9a523adca05c1f0fa95fdba9bc11671864577ff3025d3ebcbe431: Status 404 returned error can't find the container with id 5f77cdd761d9a523adca05c1f0fa95fdba9bc11671864577ff3025d3ebcbe431 Oct 31 13:30:36.966250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.966213 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:36.966349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.966282 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:36.966349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:36.966311 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:37.760097 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.760046 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" event={"ID":"6aa965c2-ad6a-4339-879f-41318554f6ce","Type":"ContainerStarted","Data":"392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc"} Oct 31 13:30:37.760771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.760154 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" event={"ID":"6aa965c2-ad6a-4339-879f-41318554f6ce","Type":"ContainerStarted","Data":"5f77cdd761d9a523adca05c1f0fa95fdba9bc11671864577ff3025d3ebcbe431"} Oct 31 13:30:37.762381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.762354 5074 generic.go:356] "Generic (PLEG): container finished" podID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerID="205be1be5222d3164a067ccf84da242ea6e8af7b96b15cd466b43831af0fe2b9" exitCode=60 Oct 31 13:30:37.762528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.762434 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerDied","Data":"205be1be5222d3164a067ccf84da242ea6e8af7b96b15cd466b43831af0fe2b9"} Oct 31 13:30:37.762528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.762498 5074 scope.go:122] "RemoveContainer" containerID="a508c190f8e4b5e7875e2d93aaf6ee974ef7164a1b77165fba7ac2bed9f24f08" Oct 31 13:30:37.762860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.762833 5074 scope.go:122] "RemoveContainer" containerID="205be1be5222d3164a067ccf84da242ea6e8af7b96b15cd466b43831af0fe2b9" Oct 31 13:30:37.763185 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:37.763148 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2_odh-ai-gateway-infra(5a23e0a1-17ab-434b-aa91-1ce660969ec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" Oct 31 13:30:37.767336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.767307 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerStarted","Data":"2dff6b6e4016eff7f359058baed93507af8d63f7393e63080d5068471d6dee02"} Oct 31 13:30:37.777685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.777430 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2"] Oct 31 13:30:37.817870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.817814 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podStartSLOduration=19.817799996 podStartE2EDuration="19.817799996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:37.815486134 +0000 UTC m=+3531.993204119" watchObservedRunningTime="2026-10-31 13:30:37.817799996 +0000 UTC m=+3531.995518030" Oct 31 13:30:37.931171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.931135 5074 scope.go:122] "RemoveContainer" containerID="b29fb08b7359e8d01e4ab3635e46756d109294dbd554da55662c69f6416bfa11" Oct 31 13:30:37.945191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.945159 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" path="/var/lib/kubelet/pods/086f29d4-9e94-45fd-a9cb-d36f19841d44/volumes" Oct 31 13:30:37.945918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.945896 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" path="/var/lib/kubelet/pods/396c4960-19d8-499d-95ce-3e5f995f07fc/volumes" Oct 31 13:30:37.946704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:37.946544 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" path="/var/lib/kubelet/pods/9a2c78e2-3449-4bef-86cd-2d09c24af2ac/volumes" Oct 31 13:30:38.681554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:38.681467 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:38.958983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:38.933438 5074 scope.go:122] "RemoveContainer" containerID="fa67278db9e2043c451c46e2048c8dfb2ce2a03226fce52163f390de7f1abfd5" Oct 31 13:30:39.164771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.157428 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:39.196329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.196274 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq"] Oct 31 13:30:39.225848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.225816 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" event={"ID":"36d64474-1d9d-49ef-84d4-487768f1ab44","Type":"ContainerStarted","Data":"90c3719dff15e3ad3b5497b55ada52f8e3329281e2d743979ecfc9b9614f9f2c"} Oct 31 13:30:39.226570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.226548 5074 scope.go:122] "RemoveContainer" containerID="90c3719dff15e3ad3b5497b55ada52f8e3329281e2d743979ecfc9b9614f9f2c" Oct 31 13:30:39.227216 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:39.227191 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb_odh-ai-gateway-infra(36d64474-1d9d-49ef-84d4-487768f1ab44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" Oct 31 13:30:39.238482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.236567 5074 generic.go:356] "Generic (PLEG): container finished" podID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerID="2dff6b6e4016eff7f359058baed93507af8d63f7393e63080d5068471d6dee02" exitCode=60 Oct 31 13:30:39.238482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.238329 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerDied","Data":"2dff6b6e4016eff7f359058baed93507af8d63f7393e63080d5068471d6dee02"} Oct 31 13:30:39.238482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.238385 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:39.238482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.238418 5074 scope.go:122] "RemoveContainer" containerID="f8c8e47bc300235c9677a1c43ddb1baf9d10e93bf808e667cff2c4ed3ea16186" Oct 31 13:30:39.240107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.240055 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:30:39.718844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.718806 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:39.721234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.721213 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:39.825595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.825500 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a23e0a1-17ab-434b-aa91-1ce660969ec0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a23e0a1-17ab-434b-aa91-1ce660969ec0-maas-api-service-ca\") pod \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " Oct 31 13:30:39.825805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.825642 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a23e0a1-17ab-434b-aa91-1ce660969ec0-kube-api-access-c9b7w\" (UniqueName: \"kubernetes.io/projected/5a23e0a1-17ab-434b-aa91-1ce660969ec0-kube-api-access-c9b7w\") pod \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\" (UID: \"5a23e0a1-17ab-434b-aa91-1ce660969ec0\") " Oct 31 13:30:39.825805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.825737 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-kube-api-access-gx2nv\" (UniqueName: \"kubernetes.io/projected/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-kube-api-access-gx2nv\") pod \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " Oct 31 13:30:39.825805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.825787 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-maas-api-service-ca\") pod \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\" (UID: \"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0\") " Oct 31 13:30:39.826862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.826824 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a23e0a1-17ab-434b-aa91-1ce660969ec0-maas-api-service-ca" pod "5a23e0a1-17ab-434b-aa91-1ce660969ec0" (UID: "5a23e0a1-17ab-434b-aa91-1ce660969ec0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:39.826983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.826868 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-maas-api-service-ca" pod "aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" (UID: "aa75011f-6c07-44fd-b5e7-7b2edda5e2e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:39.829039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.829011 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a23e0a1-17ab-434b-aa91-1ce660969ec0-kube-api-access-c9b7w" pod "5a23e0a1-17ab-434b-aa91-1ce660969ec0" (UID: "5a23e0a1-17ab-434b-aa91-1ce660969ec0"). InnerVolumeSpecName "kube-api-access-c9b7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:39.829152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.829051 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-kube-api-access-gx2nv" pod "aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" (UID: "aa75011f-6c07-44fd-b5e7-7b2edda5e2e0"). InnerVolumeSpecName "kube-api-access-gx2nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:39.929539 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.929495 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9b7w\" (UniqueName: \"kubernetes.io/projected/5a23e0a1-17ab-434b-aa91-1ce660969ec0-kube-api-access-c9b7w\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:39.929539 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.929541 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2nv\" (UniqueName: \"kubernetes.io/projected/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-kube-api-access-gx2nv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:39.929767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.929557 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:39.929767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:39.929575 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a23e0a1-17ab-434b-aa91-1ce660969ec0-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:40.265449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.265407 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" event={"ID":"aa75011f-6c07-44fd-b5e7-7b2edda5e2e0","Type":"ContainerDied","Data":"b8d06e1af094e9817830f1239a3ede911a55bebe9f1449a9244f04a6363e042d"} Oct 31 13:30:40.265961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.265481 5074 scope.go:122] "RemoveContainer" containerID="2dff6b6e4016eff7f359058baed93507af8d63f7393e63080d5068471d6dee02" Oct 31 13:30:40.265961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.265594 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq" Oct 31 13:30:40.272938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.272905 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" event={"ID":"5a23e0a1-17ab-434b-aa91-1ce660969ec0","Type":"ContainerDied","Data":"82d87c1339aafa135aeea86bbfeb3802c0ae9f4679df0bc3a9bea9c5f6e08c30"} Oct 31 13:30:40.273055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.273019 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2" Oct 31 13:30:40.276075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.276033 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerStarted","Data":"d7b6e95df2414db1b3e9d5037cf99f8009e85404bfd064a8ddf96ec1cf7f51c4"} Oct 31 13:30:40.278819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.278799 5074 generic.go:356] "Generic (PLEG): container finished" podID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerID="90c3719dff15e3ad3b5497b55ada52f8e3329281e2d743979ecfc9b9614f9f2c" exitCode=60 Oct 31 13:30:40.279391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.279366 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" event={"ID":"36d64474-1d9d-49ef-84d4-487768f1ab44","Type":"ContainerDied","Data":"90c3719dff15e3ad3b5497b55ada52f8e3329281e2d743979ecfc9b9614f9f2c"} Oct 31 13:30:40.279825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.279804 5074 scope.go:122] "RemoveContainer" containerID="90c3719dff15e3ad3b5497b55ada52f8e3329281e2d743979ecfc9b9614f9f2c" Oct 31 13:30:40.280149 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:40.280125 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb_odh-ai-gateway-infra(36d64474-1d9d-49ef-84d4-487768f1ab44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" Oct 31 13:30:40.297260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.297230 5074 scope.go:122] "RemoveContainer" containerID="205be1be5222d3164a067ccf84da242ea6e8af7b96b15cd466b43831af0fe2b9" Oct 31 13:30:40.301030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.301007 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq"] Oct 31 13:30:40.311802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.311778 5074 scope.go:122] "RemoveContainer" containerID="b29fb08b7359e8d01e4ab3635e46756d109294dbd554da55662c69f6416bfa11" Oct 31 13:30:40.311926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.311803 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5dbd39-29890890-zx9vq"] Oct 31 13:30:40.323580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.323557 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb"] Oct 31 13:30:40.441784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.441502 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2"] Oct 31 13:30:40.466877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:40.466841 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a13836-29890890-72mv2"] Oct 31 13:30:41.296292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.296259 5074 generic.go:356] "Generic (PLEG): container finished" podID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerID="d7b6e95df2414db1b3e9d5037cf99f8009e85404bfd064a8ddf96ec1cf7f51c4" exitCode=60 Oct 31 13:30:41.296775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.296336 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerDied","Data":"d7b6e95df2414db1b3e9d5037cf99f8009e85404bfd064a8ddf96ec1cf7f51c4"} Oct 31 13:30:41.296775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.296445 5074 scope.go:122] "RemoveContainer" containerID="fa67278db9e2043c451c46e2048c8dfb2ce2a03226fce52163f390de7f1abfd5" Oct 31 13:30:41.296917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.296875 5074 scope.go:122] "RemoveContainer" containerID="d7b6e95df2414db1b3e9d5037cf99f8009e85404bfd064a8ddf96ec1cf7f51c4" Oct 31 13:30:41.297271 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:41.297246 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg_odh-ai-gateway-infra(3d7495d8-6556-4de7-aac1-d560a8b63fb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" Oct 31 13:30:41.353807 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.353775 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg"] Oct 31 13:30:41.547293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.546776 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:41.694695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.694620 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d64474-1d9d-49ef-84d4-487768f1ab44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d64474-1d9d-49ef-84d4-487768f1ab44-maas-api-service-ca\") pod \"36d64474-1d9d-49ef-84d4-487768f1ab44\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " Oct 31 13:30:41.694881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.694798 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d64474-1d9d-49ef-84d4-487768f1ab44-kube-api-access-lpnvj\" (UniqueName: \"kubernetes.io/projected/36d64474-1d9d-49ef-84d4-487768f1ab44-kube-api-access-lpnvj\") pod \"36d64474-1d9d-49ef-84d4-487768f1ab44\" (UID: \"36d64474-1d9d-49ef-84d4-487768f1ab44\") " Oct 31 13:30:41.695077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.695032 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d64474-1d9d-49ef-84d4-487768f1ab44-maas-api-service-ca" pod "36d64474-1d9d-49ef-84d4-487768f1ab44" (UID: "36d64474-1d9d-49ef-84d4-487768f1ab44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:41.695780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.695751 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d64474-1d9d-49ef-84d4-487768f1ab44-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:41.698212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.698181 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d64474-1d9d-49ef-84d4-487768f1ab44-kube-api-access-lpnvj" pod "36d64474-1d9d-49ef-84d4-487768f1ab44" (UID: "36d64474-1d9d-49ef-84d4-487768f1ab44"). InnerVolumeSpecName "kube-api-access-lpnvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:41.797928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.797835 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpnvj\" (UniqueName: \"kubernetes.io/projected/36d64474-1d9d-49ef-84d4-487768f1ab44-kube-api-access-lpnvj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:41.942971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.942933 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" path="/var/lib/kubelet/pods/5a23e0a1-17ab-434b-aa91-1ce660969ec0/volumes" Oct 31 13:30:41.943509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:41.943491 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" path="/var/lib/kubelet/pods/aa75011f-6c07-44fd-b5e7-7b2edda5e2e0/volumes" Oct 31 13:30:42.313480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.313439 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr"] Oct 31 13:30:42.314122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314100 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" Oct 31 13:30:42.314486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314466 5074 cpu_manager.go:414] "containerMap: removing container" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:42.314548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314487 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:42.314548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314498 5074 cpu_manager.go:414] "containerMap: removing container" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:42.314548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314504 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:42.314548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314522 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.314548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314528 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.314548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314539 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.314548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314547 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314574 5074 cpu_manager.go:414] "containerMap: removing container" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314582 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314594 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314599 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314614 5074 cpu_manager.go:414] "containerMap: removing container" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314621 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314646 5074 cpu_manager.go:414] "containerMap: removing container" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314654 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314669 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314676 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314689 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314696 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2c78e2-3449-4bef-86cd-2d09c24af2ac" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314721 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314729 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e89d8ad-db41-4694-8db6-524a9c06496f" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314746 5074 cpu_manager.go:414] "containerMap: removing container" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314754 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314769 5074 cpu_manager.go:414] "containerMap: removing container" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314776 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314958 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314979 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="646e2652-dde7-413e-97fd-c2ee731dbe67" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.314991 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315004 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83865ad-ffe7-47f5-8a1d-fb923e0a6b97" containerName="cleanup" Oct 31 13:30:42.315003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315014 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="086f29d4-9e94-45fd-a9cb-d36f19841d44" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315024 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7357272-2464-4f33-b723-597f306797bd" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315036 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315046 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315059 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fa64c7-5eed-46d0-9771-8e5554ee2ac8" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315071 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="396c4960-19d8-499d-95ce-3e5f995f07fc" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315084 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315098 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.316235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.315110 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" containerName="cleanup" Oct 31 13:30:42.321339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.321314 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb" event={"ID":"36d64474-1d9d-49ef-84d4-487768f1ab44","Type":"ContainerDied","Data":"ea100a665ca1d5f5f5988263f7e0170038994e9e8f6a52816aecff8667ddea5e"} Oct 31 13:30:42.321474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.321351 5074 scope.go:122] "RemoveContainer" containerID="90c3719dff15e3ad3b5497b55ada52f8e3329281e2d743979ecfc9b9614f9f2c" Oct 31 13:30:42.321474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.321435 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.348194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.348161 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-09db67\"" Oct 31 13:30:42.349005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.348977 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr"] Oct 31 13:30:42.390271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.390218 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb"] Oct 31 13:30:42.407776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.407736 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-384dbe-29890890-5v8zb"] Oct 31 13:30:42.409880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.409842 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflbx\" (UniqueName: \"kubernetes.io/projected/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-kube-api-access-zflbx\") pod \"payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.418772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.410314 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.518310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.518265 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflbx\" (UniqueName: \"kubernetes.io/projected/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-kube-api-access-zflbx\") pod \"payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.518783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.518751 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.520002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.519978 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.595728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.587454 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflbx\" (UniqueName: \"kubernetes.io/projected/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-kube-api-access-zflbx\") pod \"payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.689960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.689923 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:42.700655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.699959 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:42.860319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.860277 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7495d8-6556-4de7-aac1-d560a8b63fb0-kube-api-access-fk9mm\" (UniqueName: \"kubernetes.io/projected/3d7495d8-6556-4de7-aac1-d560a8b63fb0-kube-api-access-fk9mm\") pod \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " Oct 31 13:30:42.861059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.860554 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7495d8-6556-4de7-aac1-d560a8b63fb0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7495d8-6556-4de7-aac1-d560a8b63fb0-maas-api-service-ca\") pod \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\" (UID: \"3d7495d8-6556-4de7-aac1-d560a8b63fb0\") " Oct 31 13:30:42.868101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.868069 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7495d8-6556-4de7-aac1-d560a8b63fb0-maas-api-service-ca" pod "3d7495d8-6556-4de7-aac1-d560a8b63fb0" (UID: "3d7495d8-6556-4de7-aac1-d560a8b63fb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:42.938656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:42.932840 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7495d8-6556-4de7-aac1-d560a8b63fb0-kube-api-access-fk9mm" pod "3d7495d8-6556-4de7-aac1-d560a8b63fb0" (UID: "3d7495d8-6556-4de7-aac1-d560a8b63fb0"). InnerVolumeSpecName "kube-api-access-fk9mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:43.033661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.033607 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7495d8-6556-4de7-aac1-d560a8b63fb0-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:43.033661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.033662 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9mm\" (UniqueName: \"kubernetes.io/projected/3d7495d8-6556-4de7-aac1-d560a8b63fb0-kube-api-access-fk9mm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:43.421876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.415214 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" event={"ID":"3d7495d8-6556-4de7-aac1-d560a8b63fb0","Type":"ContainerDied","Data":"1e115098b614268c1362d408147bcfa32d83bbed494b7668a8643fa7eb25c661"} Oct 31 13:30:43.427040 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.427008 5074 scope.go:122] "RemoveContainer" containerID="d7b6e95df2414db1b3e9d5037cf99f8009e85404bfd064a8ddf96ec1cf7f51c4" Oct 31 13:30:43.427287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.426737 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg" Oct 31 13:30:43.907363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.907326 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr"] Oct 31 13:30:43.925741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.925711 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg"] Oct 31 13:30:43.947869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.947782 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d64474-1d9d-49ef-84d4-487768f1ab44" path="/var/lib/kubelet/pods/36d64474-1d9d-49ef-84d4-487768f1ab44/volumes" Oct 31 13:30:43.948386 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:43.948361 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ef744-29890890-7mhdg"] Oct 31 13:30:44.630517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:44.630474 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" event={"ID":"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04","Type":"ContainerStarted","Data":"7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261"} Oct 31 13:30:44.631008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:44.630547 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" event={"ID":"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04","Type":"ContainerStarted","Data":"bb1b67d5b169cad4f4da05191a46e2e6c42e7b5daa6d360e799bf51644ebc9c4"} Oct 31 13:30:44.631008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:44.630575 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:44.671518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:44.671464 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" podStartSLOduration=2.671448857 podStartE2EDuration="2.671448857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:44.66300875 +0000 UTC m=+3538.840726734" watchObservedRunningTime="2026-10-31 13:30:44.671448857 +0000 UTC m=+3538.849166840" Oct 31 13:30:45.357259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.357214 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6"] Oct 31 13:30:45.358217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358187 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:30:45.358217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358213 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:30:45.358427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358230 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:30:45.358427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358235 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:30:45.358427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358303 5074 cpu_manager.go:414] "containerMap: removing container" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:45.358427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358309 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:45.358667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358432 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa75011f-6c07-44fd-b5e7-7b2edda5e2e0" containerName="cleanup" Oct 31 13:30:45.358667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358443 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:30:45.358667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358450 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a23e0a1-17ab-434b-aa91-1ce660969ec0" containerName="cleanup" Oct 31 13:30:45.358667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.358458 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:30:45.362771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.362738 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.385962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.385923 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6"] Oct 31 13:30:45.387255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.387227 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3cddb3\"" Oct 31 13:30:45.451458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.451419 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec7097-1544-40c6-92c0-14c712036295-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.451715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.451603 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcb4d\" (UniqueName: \"kubernetes.io/projected/97ec7097-1544-40c6-92c0-14c712036295-kube-api-access-wcb4d\") pod \"payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.574655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.574540 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec7097-1544-40c6-92c0-14c712036295-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.574846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.574715 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcb4d\" (UniqueName: \"kubernetes.io/projected/97ec7097-1544-40c6-92c0-14c712036295-kube-api-access-wcb4d\") pod \"payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.622699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.611742 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcb4d\" (UniqueName: \"kubernetes.io/projected/97ec7097-1544-40c6-92c0-14c712036295-kube-api-access-wcb4d\") pod \"payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.641716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.640701 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec7097-1544-40c6-92c0-14c712036295-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.702338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.701002 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:45.985066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:45.985020 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" path="/var/lib/kubelet/pods/3d7495d8-6556-4de7-aac1-d560a8b63fb0/volumes" Oct 31 13:30:46.349734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:46.348594 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6"] Oct 31 13:30:46.350695 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:30:46.350666 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ec7097_1544_40c6_92c0_14c712036295.slice/crio-ac58c30551a0cd93c31fb2d77b35b525c1b2f1b55441b4f1f76f5ad9fd489884 WatchSource:0}: Error finding container ac58c30551a0cd93c31fb2d77b35b525c1b2f1b55441b4f1f76f5ad9fd489884: Status 404 returned error can't find the container with id ac58c30551a0cd93c31fb2d77b35b525c1b2f1b55441b4f1f76f5ad9fd489884 Oct 31 13:30:46.872385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:46.872348 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" event={"ID":"97ec7097-1544-40c6-92c0-14c712036295","Type":"ContainerStarted","Data":"47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64"} Oct 31 13:30:46.872938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:46.872416 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" event={"ID":"97ec7097-1544-40c6-92c0-14c712036295","Type":"ContainerStarted","Data":"ac58c30551a0cd93c31fb2d77b35b525c1b2f1b55441b4f1f76f5ad9fd489884"} Oct 31 13:30:46.872938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:46.872444 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:30:46.897172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:46.897115 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" podStartSLOduration=1.89710152 podStartE2EDuration="1.89710152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:46.893686378 +0000 UTC m=+3541.071404366" watchObservedRunningTime="2026-10-31 13:30:46.89710152 +0000 UTC m=+3541.074819504" Oct 31 13:30:55.896142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:55.887593 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:30:55.994166 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:30:55.994119 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:30:57.889069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:30:57.889039 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:31:12.255210 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:31:12.255172 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]" Oct 31 13:31:24.032002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:24.031969 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5"] Oct 31 13:31:24.038649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:24.038610 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247da670-j9nw5"] Oct 31 13:31:24.904849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:24.904807 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:31:25.943364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:25.943328 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7abb12e-61a4-42bc-ae37-e6120396e413" path="/var/lib/kubelet/pods/a7abb12e-61a4-42bc-ae37-e6120396e413/volumes" Oct 31 13:31:25.986289 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:31:25.986255 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:31:34.113977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:34.113928 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z"] Oct 31 13:31:34.125044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:34.125011 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe777c-4kd2z"] Oct 31 13:31:35.977766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:35.977731 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eec3c22-4741-4d2d-b198-471a1574fb9f" path="/var/lib/kubelet/pods/0eec3c22-4741-4d2d-b198-471a1574fb9f/volumes" Oct 31 13:31:45.469551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:45.469506 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh"] Oct 31 13:31:45.470746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:45.470673 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" containerID="cri-o://020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856" gracePeriod=30 Oct 31 13:31:47.049694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:47.049659 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:31:47.066375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:47.066350 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:31:50.953067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:50.953038 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:31:51.047579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.047541 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-data\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.047845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.047599 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-socket\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.047845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.047711 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-kube-api-access-qqrct\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-kube-api-access-qqrct\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.047845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.047826 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-envoy\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.048048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.047855 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-credential-socket\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.048048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.047962 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784ac98e-380a-450d-834f-fa8bcd548b59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784ac98e-380a-450d-834f-fa8bcd548b59-istiod-ca-cert\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.048048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.048011 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-socket" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:51.048219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.048129 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-istio-token\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-istio-token\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.048317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.048295 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-data" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:51.048388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.048303 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/784ac98e-380a-450d-834f-fa8bcd548b59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784ac98e-380a-450d-834f-fa8bcd548b59-istio-podinfo\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.048388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.048367 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784ac98e-380a-450d-834f-fa8bcd548b59-istiod-ca-cert" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:51.048388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.048379 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-certs\") pod \"784ac98e-380a-450d-834f-fa8bcd548b59\" (UID: \"784ac98e-380a-450d-834f-fa8bcd548b59\") " Oct 31 13:31:51.048976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.048947 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-credential-socket" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:51.049101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.049078 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-certs" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:51.050074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.050047 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.050181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.050080 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.050181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.050097 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.050181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.050129 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784ac98e-380a-450d-834f-fa8bcd548b59-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.050181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.050145 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.052497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.052459 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-envoy" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:51.052876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.052852 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-kube-api-access-qqrct" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "kube-api-access-qqrct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:51.053207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.053183 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-istio-token" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:51.053517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.053261 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/784ac98e-380a-450d-834f-fa8bcd548b59-istio-podinfo" pod "784ac98e-380a-450d-834f-fa8bcd548b59" (UID: "784ac98e-380a-450d-834f-fa8bcd548b59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:51.151869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.151833 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrct\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-kube-api-access-qqrct\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.151869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.151867 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784ac98e-380a-450d-834f-fa8bcd548b59-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.151869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.151877 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784ac98e-380a-450d-834f-fa8bcd548b59-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.152118 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.151886 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784ac98e-380a-450d-834f-fa8bcd548b59-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:51.796687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.779978 5074 generic.go:356] "Generic (PLEG): container finished" podID="784ac98e-380a-450d-834f-fa8bcd548b59" containerID="020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856" exitCode=0 Oct 31 13:31:51.796687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.780084 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" event={"ID":"784ac98e-380a-450d-834f-fa8bcd548b59","Type":"ContainerDied","Data":"020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856"} Oct 31 13:31:51.796687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.780138 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" event={"ID":"784ac98e-380a-450d-834f-fa8bcd548b59","Type":"ContainerDied","Data":"b84e883da0d87e2072e699eec406884188d3b09e11b098608ecd05f13a2dd90c"} Oct 31 13:31:51.796687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.780167 5074 scope.go:122] "RemoveContainer" containerID="020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856" Oct 31 13:31:51.796687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:51.780349 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh" Oct 31 13:31:52.007530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:52.007497 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh"] Oct 31 13:31:52.019164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:52.019133 5074 scope.go:122] "RemoveContainer" containerID="020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856" Oct 31 13:31:52.019933 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:31:52.019903 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856\": container with ID starting with 020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856 not found: ID does not exist" containerID="020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856" Oct 31 13:31:52.020183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:52.020113 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856"} err="failed to get container status \"020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856\": rpc error: code = NotFound desc = could not find container \"020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856\": container with ID starting with 020eb68fc6a2b0a9ac5ed1cbd70c288f6d3c13d52a43265ef512c13c2c73a856 not found: ID does not exist" Oct 31 13:31:52.049252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:52.049144 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-mp5fh"] Oct 31 13:31:52.132123 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:31:52.132071 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784ac98e_380a_450d_834f_fa8bcd548b59.slice/crio-b84e883da0d87e2072e699eec406884188d3b09e11b098608ecd05f13a2dd90c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784ac98e_380a_450d_834f_fa8bcd548b59.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:31:54.135333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:54.135006 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" path="/var/lib/kubelet/pods/784ac98e-380a-450d-834f-fa8bcd548b59/volumes" Oct 31 13:31:54.406715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:54.402287 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt"] Oct 31 13:31:55.256077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:55.254841 5074 scope.go:122] "RemoveContainer" containerID="f71185d810d3966273f0b7598d92eddc8407a279cc37d9b4aa477d158e8c75ac" Oct 31 13:31:55.306584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:55.306544 5074 scope.go:122] "RemoveContainer" containerID="4f4e38bae3cfa66e2ac8fcf10ff42f54241d62f1ff4ff542820940e67809915d" Oct 31 13:31:55.827657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:55.827543 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh"] Oct 31 13:31:55.829700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:55.828144 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" containerID="cri-o://87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf" gracePeriod=30 Oct 31 13:31:55.834083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:55.834054 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz"] Oct 31 13:31:55.834242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:55.834223 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" containerID="cri-o://2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e" gracePeriod=30 Oct 31 13:31:56.353383 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:31:56.353302 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:31:59.403859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:31:59.403809 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" podUID="d772dc21-8fd6-43a0-ba9d-be8854db5729" containerName="payload-pre-processing" containerID="cri-o://c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7" gracePeriod=25 Oct 31 13:32:00.698664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:00.683744 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" podUID="d772dc21-8fd6-43a0-ba9d-be8854db5729" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:01.290662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.288157 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:32:01.347836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.347798 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:32:01.389470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.389415 5074 generic.go:356] "Generic (PLEG): container finished" podID="cc8ad859-e258-4a60-b804-ab05271038b5" containerID="87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf" exitCode=0 Oct 31 13:32:01.389755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.389710 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" event={"ID":"cc8ad859-e258-4a60-b804-ab05271038b5","Type":"ContainerDied","Data":"87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf"} Oct 31 13:32:01.389851 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.389812 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" event={"ID":"cc8ad859-e258-4a60-b804-ab05271038b5","Type":"ContainerDied","Data":"242be2c15a3b835422d4cd3c864b4f526a5f8d99354993aa2e413a6d996ea02a"} Oct 31 13:32:01.389920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.389850 5074 scope.go:122] "RemoveContainer" containerID="87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf" Oct 31 13:32:01.390106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.390086 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh" Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.429979 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-envoy\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430067 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-credential-socket\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430135 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-socket\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430194 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-data\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430283 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-certs\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430426 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-kube-api-access-66wb7\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-kube-api-access-66wb7\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430485 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8ad859-e258-4a60-b804-ab05271038b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8ad859-e258-4a60-b804-ab05271038b5-istiod-ca-cert\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430552 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-istio-token\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-istio-token\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.430704 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc8ad859-e258-4a60-b804-ab05271038b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8ad859-e258-4a60-b804-ab05271038b5-istio-podinfo\") pod \"cc8ad859-e258-4a60-b804-ab05271038b5\" (UID: \"cc8ad859-e258-4a60-b804-ab05271038b5\") " Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.433394 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-credential-socket" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.433762 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-socket" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.433922 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8ad859-e258-4a60-b804-ab05271038b5-istiod-ca-cert" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.434187 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-certs" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.434754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.434295 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-data" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.435660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.435195 5074 generic.go:356] "Generic (PLEG): container finished" podID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerID="2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e" exitCode=0 Oct 31 13:32:01.435660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.435339 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" Oct 31 13:32:01.435660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.435392 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" event={"ID":"9083d229-6ab4-4749-bfa7-d69cab443a32","Type":"ContainerDied","Data":"2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e"} Oct 31 13:32:01.435660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.435448 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz" event={"ID":"9083d229-6ab4-4749-bfa7-d69cab443a32","Type":"ContainerDied","Data":"a30fb6f848f43ba94315f2e5e2d89e235b2cfbeb9ecc6911d25669b6b6d867b9"} Oct 31 13:32:01.439150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.438502 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-envoy" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.439150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.438808 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-istio-token" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.439150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.438842 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-kube-api-access-66wb7" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "kube-api-access-66wb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.439876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.439852 5074 scope.go:122] "RemoveContainer" containerID="ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705" Oct 31 13:32:01.440398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.440334 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc8ad859-e258-4a60-b804-ab05271038b5-istio-podinfo" pod "cc8ad859-e258-4a60-b804-ab05271038b5" (UID: "cc8ad859-e258-4a60-b804-ab05271038b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:01.454992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.454910 5074 scope.go:122] "RemoveContainer" containerID="87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf" Oct 31 13:32:01.455246 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:01.455215 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf\": container with ID starting with 87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf not found: ID does not exist" containerID="87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf" Oct 31 13:32:01.455319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.455257 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf"} err="failed to get container status \"87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf\": rpc error: code = NotFound desc = could not find container \"87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf\": container with ID starting with 87dfcd03066f8b315cf5dcbcd101f6f94b26cc16eb26ee6a8e26d6c6a76037cf not found: ID does not exist" Oct 31 13:32:01.455319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.455280 5074 scope.go:122] "RemoveContainer" containerID="ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705" Oct 31 13:32:01.455924 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:01.455718 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705\": container with ID starting with ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705 not found: ID does not exist" containerID="ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705" Oct 31 13:32:01.455924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.455747 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705"} err="failed to get container status \"ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705\": rpc error: code = NotFound desc = could not find container \"ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705\": container with ID starting with ca4c09f9f0a7611fae72c8f9a4d9134988fccbd2d2984d4c8469bad63c7bf705 not found: ID does not exist" Oct 31 13:32:01.455924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.455770 5074 scope.go:122] "RemoveContainer" containerID="2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e" Oct 31 13:32:01.468553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.468342 5074 scope.go:122] "RemoveContainer" containerID="2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e" Oct 31 13:32:01.468907 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:01.468816 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e\": container with ID starting with 2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e not found: ID does not exist" containerID="2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e" Oct 31 13:32:01.468907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.468850 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e"} err="failed to get container status \"2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e\": rpc error: code = NotFound desc = could not find container \"2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e\": container with ID starting with 2169aef4c1954ee0c3f8c9c3e1d46f643a5835fb167a6878318941a5847f218e not found: ID does not exist" Oct 31 13:32:01.532203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532160 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-socket\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532306 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-token\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-token\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532346 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-credential-socket\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532435 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-podinfo\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532487 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-socket" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.532675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532584 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-certs\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532665 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-kube-api-access-7pzgf\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-kube-api-access-7pzgf\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532742 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9083d229-6ab4-4749-bfa7-d69cab443a32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9083d229-6ab4-4749-bfa7-d69cab443a32-istiod-ca-cert\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532755 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-credential-socket" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.532801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532794 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-data\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.532972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.532849 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-envoy\") pod \"9083d229-6ab4-4749-bfa7-d69cab443a32\" (UID: \"9083d229-6ab4-4749-bfa7-d69cab443a32\") " Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533829 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533834 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-certs" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533858 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8ad859-e258-4a60-b804-ab05271038b5-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533891 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533914 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533930 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533945 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533960 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533973 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8ad859-e258-4a60-b804-ab05271038b5-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533981 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9083d229-6ab4-4749-bfa7-d69cab443a32-istiod-ca-cert" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.533987 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.534021 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wb7\" (UniqueName: \"kubernetes.io/projected/cc8ad859-e258-4a60-b804-ab05271038b5-kube-api-access-66wb7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.534171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.534049 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8ad859-e258-4a60-b804-ab05271038b5-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.535092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.534217 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-data" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.535697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.535656 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-kube-api-access-7pzgf" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "kube-api-access-7pzgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.535817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.535725 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-token" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:01.536151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.536121 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-podinfo" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:01.536984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.536962 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-envoy" pod "9083d229-6ab4-4749-bfa7-d69cab443a32" (UID: "9083d229-6ab4-4749-bfa7-d69cab443a32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:01.635402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.635366 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.635402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.635401 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.635402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.635410 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.635402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.635421 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pzgf\" (UniqueName: \"kubernetes.io/projected/9083d229-6ab4-4749-bfa7-d69cab443a32-kube-api-access-7pzgf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.635719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.635433 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9083d229-6ab4-4749-bfa7-d69cab443a32-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.635719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.635443 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.635719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.635452 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9083d229-6ab4-4749-bfa7-d69cab443a32-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:01.798654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.787438 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh"] Oct 31 13:32:01.804452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.804400 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-sz4sh"] Oct 31 13:32:01.862981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.862899 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz"] Oct 31 13:32:01.867860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.867828 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dbd39-openshift-default-6d9d4858cc-5vhnz"] Oct 31 13:32:01.944271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.944233 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" path="/var/lib/kubelet/pods/9083d229-6ab4-4749-bfa7-d69cab443a32/volumes" Oct 31 13:32:01.944795 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:01.944774 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" path="/var/lib/kubelet/pods/cc8ad859-e258-4a60-b804-ab05271038b5/volumes" Oct 31 13:32:06.548450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:06.548411 5074 generic.go:356] "Generic (PLEG): container finished" podID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerID="392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc" exitCode=0 Oct 31 13:32:06.549304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:06.549278 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" event={"ID":"6aa965c2-ad6a-4339-879f-41318554f6ce","Type":"ContainerDied","Data":"392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc"} Oct 31 13:32:06.549972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:06.549951 5074 scope.go:122] "RemoveContainer" containerID="392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc" Oct 31 13:32:06.557729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:06.557697 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:06.557939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:06.557920 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:06.558087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:06.558070 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:07.563110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:07.563066 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" event={"ID":"6aa965c2-ad6a-4339-879f-41318554f6ce","Type":"ContainerStarted","Data":"1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799"} Oct 31 13:32:07.655187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:07.655149 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:32:07.656408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:07.656372 5074 patch_prober.go:29] interesting pod/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 31 13:32:07.656545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:07.656432 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 31 13:32:08.654481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:08.654449 5074 patch_prober.go:29] interesting pod/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 31 13:32:08.654889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:08.654501 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 31 13:32:09.246053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:09.239726 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:32:09.654989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:09.654903 5074 patch_prober.go:29] interesting pod/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 31 13:32:09.654989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:09.654962 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 31 13:32:10.653615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:10.653566 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" podUID="d772dc21-8fd6-43a0-ba9d-be8854db5729" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:10.654755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:10.654729 5074 patch_prober.go:29] interesting pod/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 31 13:32:10.654887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:10.654773 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 31 13:32:11.654977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:11.654918 5074 patch_prober.go:29] interesting pod/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 31 13:32:11.655395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:11.654998 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 31 13:32:12.167269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:12.167226 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk"] Oct 31 13:32:12.167532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:12.167484 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" containerID="cri-o://690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277" gracePeriod=30 Oct 31 13:32:12.659655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:12.655369 5074 patch_prober.go:29] interesting pod/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 31 13:32:12.659655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:12.655431 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 31 13:32:13.654707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:13.654620 5074 patch_prober.go:29] interesting pod/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Oct 31 13:32:13.654901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:13.654728 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Oct 31 13:32:14.677117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:14.665173 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:32:14.680514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:14.680472 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:32:17.492951 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.492913 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:32:17.517975 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.517937 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-data\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518128 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-socket\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518204 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-data" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.518289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518255 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12278131-c5a5-4c02-8544-7a943a5c93bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12278131-c5a5-4c02-8544-7a943a5c93bd-istiod-ca-cert\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518348 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-token\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-token\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518357 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-socket" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.518580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518560 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-kube-api-access-pkmtt\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-kube-api-access-pkmtt\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518622 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-credential-socket\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518708 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12278131-c5a5-4c02-8544-7a943a5c93bd-istiod-ca-cert" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:17.518791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518723 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-certs\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518845 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-credential-socket" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.518933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518770 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-podinfo\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.518994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518948 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-certs" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.518994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.518962 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-envoy\") pod \"12278131-c5a5-4c02-8544-7a943a5c93bd\" (UID: \"12278131-c5a5-4c02-8544-7a943a5c93bd\") " Oct 31 13:32:17.519895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.519866 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.519895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.519897 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.520111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.519914 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.520111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.519928 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.520111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.519941 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12278131-c5a5-4c02-8544-7a943a5c93bd-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.524669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.521769 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-podinfo" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:32:17.524669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.521826 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-kube-api-access-pkmtt" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "kube-api-access-pkmtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:17.524669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.521854 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-token" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:17.524669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.522140 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-envoy" pod "12278131-c5a5-4c02-8544-7a943a5c93bd" (UID: "12278131-c5a5-4c02-8544-7a943a5c93bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:32:17.622340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.622288 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkmtt\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-kube-api-access-pkmtt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.622340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.622332 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.622340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.622346 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.622610 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.622358 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12278131-c5a5-4c02-8544-7a943a5c93bd-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:17.814479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.814437 5074 generic.go:356] "Generic (PLEG): container finished" podID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerID="690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277" exitCode=0 Oct 31 13:32:17.814731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.814495 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" Oct 31 13:32:17.814731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.814494 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" event={"ID":"12278131-c5a5-4c02-8544-7a943a5c93bd","Type":"ContainerDied","Data":"690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277"} Oct 31 13:32:17.814731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.814615 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk" event={"ID":"12278131-c5a5-4c02-8544-7a943a5c93bd","Type":"ContainerDied","Data":"c0b076dcabb780729e726facdd445c161f24dc6620a2b43bb5e2d9ed3795e857"} Oct 31 13:32:17.814731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.814663 5074 scope.go:122] "RemoveContainer" containerID="690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277" Oct 31 13:32:17.827866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.827845 5074 scope.go:122] "RemoveContainer" containerID="690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277" Oct 31 13:32:17.828178 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:17.828157 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277\": container with ID starting with 690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277 not found: ID does not exist" containerID="690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277" Oct 31 13:32:17.828244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.828185 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277"} err="failed to get container status \"690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277\": rpc error: code = NotFound desc = could not find container \"690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277\": container with ID starting with 690a5b7315aca6d32ac1a4ded0af84703d6d395476e3350910cee59b8ce7c277 not found: ID does not exist" Oct 31 13:32:17.847623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.847587 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk"] Oct 31 13:32:17.857140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.857107 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4023ad-openshift-default-55ffb99979-k89jk"] Oct 31 13:32:17.941850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:17.941814 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" path="/var/lib/kubelet/pods/12278131-c5a5-4c02-8544-7a943a5c93bd/volumes" Oct 31 13:32:18.388620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:18.388580 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5"] Oct 31 13:32:18.397879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:18.397842 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq"] Oct 31 13:32:19.746053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.746011 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:32:19.807684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.807647 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d772dc21-8fd6-43a0-ba9d-be8854db5729-kube-api-access-hn5vx\" (UniqueName: \"kubernetes.io/projected/d772dc21-8fd6-43a0-ba9d-be8854db5729-kube-api-access-hn5vx\") pod \"d772dc21-8fd6-43a0-ba9d-be8854db5729\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " Oct 31 13:32:19.808252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.808225 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d772dc21-8fd6-43a0-ba9d-be8854db5729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d772dc21-8fd6-43a0-ba9d-be8854db5729-plugins-config-volume\") pod \"d772dc21-8fd6-43a0-ba9d-be8854db5729\" (UID: \"d772dc21-8fd6-43a0-ba9d-be8854db5729\") " Oct 31 13:32:19.809461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.809436 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d772dc21-8fd6-43a0-ba9d-be8854db5729-plugins-config-volume" pod "d772dc21-8fd6-43a0-ba9d-be8854db5729" (UID: "d772dc21-8fd6-43a0-ba9d-be8854db5729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:19.810614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.810576 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d772dc21-8fd6-43a0-ba9d-be8854db5729-kube-api-access-hn5vx" pod "d772dc21-8fd6-43a0-ba9d-be8854db5729" (UID: "d772dc21-8fd6-43a0-ba9d-be8854db5729"). InnerVolumeSpecName "kube-api-access-hn5vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:19.971926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.971892 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d772dc21-8fd6-43a0-ba9d-be8854db5729-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:19.972108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.972097 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5vx\" (UniqueName: \"kubernetes.io/projected/d772dc21-8fd6-43a0-ba9d-be8854db5729-kube-api-access-hn5vx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:19.974807 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.974771 5074 generic.go:356] "Generic (PLEG): container finished" podID="d772dc21-8fd6-43a0-ba9d-be8854db5729" containerID="c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7" exitCode=0 Oct 31 13:32:19.974986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.974929 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" Oct 31 13:32:19.982946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.982905 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" event={"ID":"d772dc21-8fd6-43a0-ba9d-be8854db5729","Type":"ContainerDied","Data":"c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7"} Oct 31 13:32:19.983192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.983172 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt" event={"ID":"d772dc21-8fd6-43a0-ba9d-be8854db5729","Type":"ContainerDied","Data":"f1907255b2d44a16bed247b83869b852615b7328634d988379df8510f53ac176"} Oct 31 13:32:19.983336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.983245 5074 scope.go:122] "RemoveContainer" containerID="c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7" Oct 31 13:32:19.999738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:19.999705 5074 scope.go:122] "RemoveContainer" containerID="c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7" Oct 31 13:32:20.000112 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:20.000088 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7\": container with ID starting with c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7 not found: ID does not exist" containerID="c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7" Oct 31 13:32:20.000190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:20.000127 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7"} err="failed to get container status \"c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7\": rpc error: code = NotFound desc = could not find container \"c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7\": container with ID starting with c03f2fdf8e552f058a5e637e2c538f5753911d610195f622ce0bc15eb3fa51b7 not found: ID does not exist" Oct 31 13:32:20.020620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:20.020574 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt"] Oct 31 13:32:20.025113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:20.025085 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4023ad-564f587b59-chqkt"] Oct 31 13:32:21.585712 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:21.585665 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:32:21.979290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:21.979255 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d772dc21-8fd6-43a0-ba9d-be8854db5729" path="/var/lib/kubelet/pods/d772dc21-8fd6-43a0-ba9d-be8854db5729/volumes" Oct 31 13:32:23.394729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:23.394685 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" podUID="1285e99f-1285-4dd7-acf8-ba979766bc74" containerName="payload-processing" containerID="cri-o://2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122" gracePeriod=25 Oct 31 13:32:23.399168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:23.398449 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" podUID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" containerName="payload-pre-processing" containerID="cri-o://8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507" gracePeriod=25 Oct 31 13:32:25.254653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:25.254603 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" podUID="1285e99f-1285-4dd7-acf8-ba979766bc74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:25.255031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:25.254620 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" podUID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:25.988196 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:25.988131 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:32:35.253680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:35.253603 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" podUID="1285e99f-1285-4dd7-acf8-ba979766bc74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:35.254181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:35.254040 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" podUID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:32:43.834583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:43.834136 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:32:43.851993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:43.851089 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-plugins-config-volume\") pod \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " Oct 31 13:32:43.852205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:43.852169 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-plugins-config-volume" pod "c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" (UID: "c4fb9ea5-b8e8-4818-9b6b-7105d343f65b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:43.862958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:43.862924 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:32:43.953432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:43.953394 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-kube-api-access-nkk4s\" (UniqueName: \"kubernetes.io/projected/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-kube-api-access-nkk4s\") pod \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\" (UID: \"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b\") " Oct 31 13:32:43.954219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:43.954191 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:43.955913 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:43.955885 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-kube-api-access-nkk4s" pod "c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" (UID: "c4fb9ea5-b8e8-4818-9b6b-7105d343f65b"). InnerVolumeSpecName "kube-api-access-nkk4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:44.055357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.055262 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1285e99f-1285-4dd7-acf8-ba979766bc74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1285e99f-1285-4dd7-acf8-ba979766bc74-plugins-config-volume\") pod \"1285e99f-1285-4dd7-acf8-ba979766bc74\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " Oct 31 13:32:44.055357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.055343 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1285e99f-1285-4dd7-acf8-ba979766bc74-kube-api-access-7hcqm\" (UniqueName: \"kubernetes.io/projected/1285e99f-1285-4dd7-acf8-ba979766bc74-kube-api-access-7hcqm\") pod \"1285e99f-1285-4dd7-acf8-ba979766bc74\" (UID: \"1285e99f-1285-4dd7-acf8-ba979766bc74\") " Oct 31 13:32:44.055713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.055688 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1285e99f-1285-4dd7-acf8-ba979766bc74-plugins-config-volume" pod "1285e99f-1285-4dd7-acf8-ba979766bc74" (UID: "1285e99f-1285-4dd7-acf8-ba979766bc74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:44.055958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.055940 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1285e99f-1285-4dd7-acf8-ba979766bc74-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:44.056017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.055963 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkk4s\" (UniqueName: \"kubernetes.io/projected/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b-kube-api-access-nkk4s\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:44.057571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.057546 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1285e99f-1285-4dd7-acf8-ba979766bc74-kube-api-access-7hcqm" pod "1285e99f-1285-4dd7-acf8-ba979766bc74" (UID: "1285e99f-1285-4dd7-acf8-ba979766bc74"). InnerVolumeSpecName "kube-api-access-7hcqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:44.157079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.157044 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hcqm\" (UniqueName: \"kubernetes.io/projected/1285e99f-1285-4dd7-acf8-ba979766bc74-kube-api-access-7hcqm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:44.422554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.421347 5074 generic.go:356] "Generic (PLEG): container finished" podID="1285e99f-1285-4dd7-acf8-ba979766bc74" containerID="2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122" exitCode=0 Oct 31 13:32:44.422554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.421568 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" event={"ID":"1285e99f-1285-4dd7-acf8-ba979766bc74","Type":"ContainerDied","Data":"2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122"} Oct 31 13:32:44.422554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.421697 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" event={"ID":"1285e99f-1285-4dd7-acf8-ba979766bc74","Type":"ContainerDied","Data":"a2313fc6319c54e3d99743d225c6c70e9dc1a81c5e160719e4423e25d2829e8c"} Oct 31 13:32:44.422554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.422127 5074 scope.go:122] "RemoveContainer" containerID="2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122" Oct 31 13:32:44.424537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.424508 5074 generic.go:356] "Generic (PLEG): container finished" podID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" containerID="8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507" exitCode=0 Oct 31 13:32:44.424707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.424620 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" event={"ID":"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b","Type":"ContainerDied","Data":"8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507"} Oct 31 13:32:44.424804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.424708 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" event={"ID":"c4fb9ea5-b8e8-4818-9b6b-7105d343f65b","Type":"ContainerDied","Data":"b3091e09d4d6e111547dfcbbf436cb116d33e22e49026afc92c17b1742df219c"} Oct 31 13:32:44.424804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.424802 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq" Oct 31 13:32:44.426797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.426770 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5" Oct 31 13:32:44.467053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.467021 5074 scope.go:122] "RemoveContainer" containerID="2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122" Oct 31 13:32:44.472027 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:44.472002 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122\": container with ID starting with 2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122 not found: ID does not exist" containerID="2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122" Oct 31 13:32:44.472219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.472188 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122"} err="failed to get container status \"2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122\": rpc error: code = NotFound desc = could not find container \"2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122\": container with ID starting with 2444cedf1cefc31a861a609dc978bd91f141a39f79146bf90dcff49cfe783122 not found: ID does not exist" Oct 31 13:32:44.472330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.472319 5074 scope.go:122] "RemoveContainer" containerID="8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507" Oct 31 13:32:44.483844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.483821 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq"] Oct 31 13:32:44.522861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.522828 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6c445-cd5c6787-fh4pq"] Oct 31 13:32:44.529897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.529866 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5"] Oct 31 13:32:44.537069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.536803 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c6c445-744c57dcb5-h5vb5"] Oct 31 13:32:44.539691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.539531 5074 scope.go:122] "RemoveContainer" containerID="8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507" Oct 31 13:32:44.540149 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:44.540126 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507\": container with ID starting with 8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507 not found: ID does not exist" containerID="8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507" Oct 31 13:32:44.540242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:44.540159 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507"} err="failed to get container status \"8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507\": rpc error: code = NotFound desc = could not find container \"8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507\": container with ID starting with 8f47cf3f38dd5fd30b21887daa933392ef1c7f35de5ff266c693d6ddf0367507 not found: ID does not exist" Oct 31 13:32:45.942677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:45.942645 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1285e99f-1285-4dd7-acf8-ba979766bc74" path="/var/lib/kubelet/pods/1285e99f-1285-4dd7-acf8-ba979766bc74/volumes" Oct 31 13:32:45.943150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:45.943135 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" path="/var/lib/kubelet/pods/c4fb9ea5-b8e8-4818-9b6b-7105d343f65b/volumes" Oct 31 13:32:55.449792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:55.449756 5074 scope.go:122] "RemoveContainer" containerID="d8751971e63afea0557bb1f4ef0eefc8389cf780c7dcdaa5038399345748d3f6" Oct 31 13:32:55.463745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:55.463716 5074 scope.go:122] "RemoveContainer" containerID="0e7f2370786c50f1085e61a282f4693432eeb0bb73ab05cf8ef30ffdd9748fcc" Oct 31 13:32:55.985605 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:32:55.985567 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:32:59.119745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:59.119712 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d"] Oct 31 13:32:59.124278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:59.124247 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ae9733-rtn8d"] Oct 31 13:32:59.952605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:32:59.951484 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d23f63d-0090-49a8-a3d0-b2cc9706096a" path="/var/lib/kubelet/pods/7d23f63d-0090-49a8-a3d0-b2cc9706096a/volumes" Oct 31 13:33:10.027330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:10.027287 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c"] Oct 31 13:33:10.031397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:10.031371 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceca5bf8-5kf6c"] Oct 31 13:33:11.975681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:11.961162 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236819fa-e461-49bd-aab5-f01761c30ec8" path="/var/lib/kubelet/pods/236819fa-e461-49bd-aab5-f01761c30ec8/volumes" Oct 31 13:33:22.027026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:22.026978 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9"] Oct 31 13:33:22.031283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:22.031250 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b197f1c-t9cb9"] Oct 31 13:33:23.940803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:23.940764 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540edb2f-2875-4b02-99ca-c9afc1161d6a" path="/var/lib/kubelet/pods/540edb2f-2875-4b02-99ca-c9afc1161d6a/volumes" Oct 31 13:33:26.051412 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:33:26.051369 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:33:32.065758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:32.065710 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl"] Oct 31 13:33:32.124270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:32.124233 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6708dd1-zgzcl"] Oct 31 13:33:33.941438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:33.941405 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305" path="/var/lib/kubelet/pods/8b6f9f30-18c9-47e8-8fa5-ae4de2cd2305/volumes" Oct 31 13:33:37.029480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:33:37.029441 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:33:55.984908 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:33:55.984858 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:34:25.985441 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:34:25.985400 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:34:32.655924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.655886 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr"] Oct 31 13:34:32.656885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656859 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656884 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656910 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656918 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656941 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1285e99f-1285-4dd7-acf8-ba979766bc74" containerName="payload-processing" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656948 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1285e99f-1285-4dd7-acf8-ba979766bc74" containerName="payload-processing" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656970 5074 cpu_manager.go:414] "containerMap: removing container" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656977 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.656998 5074 cpu_manager.go:414] "containerMap: removing container" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" Oct 31 13:34:32.657020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657005 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657026 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657033 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657065 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657073 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657096 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" containerName="payload-pre-processing" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657104 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" containerName="payload-pre-processing" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657120 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d772dc21-8fd6-43a0-ba9d-be8854db5729" containerName="payload-pre-processing" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657130 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d772dc21-8fd6-43a0-ba9d-be8854db5729" containerName="payload-pre-processing" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657312 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657328 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fb9ea5-b8e8-4818-9b6b-7105d343f65b" containerName="payload-pre-processing" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657341 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d772dc21-8fd6-43a0-ba9d-be8854db5729" containerName="payload-pre-processing" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657357 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9083d229-6ab4-4749-bfa7-d69cab443a32" containerName="istio-proxy" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657370 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1285e99f-1285-4dd7-acf8-ba979766bc74" containerName="payload-processing" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657383 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7495d8-6556-4de7-aac1-d560a8b63fb0" containerName="cleanup" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657392 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="784ac98e-380a-450d-834f-fa8bcd548b59" containerName="istio-proxy" Oct 31 13:34:32.657505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.657402 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="12278131-c5a5-4c02-8544-7a943a5c93bd" containerName="istio-proxy" Oct 31 13:34:32.661859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.661838 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.667512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.666943 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:34:32.667512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.667008 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:34:32.671355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.671330 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr"] Oct 31 13:34:32.812074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.812037 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.812291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.812240 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvgs\" (UniqueName: \"kubernetes.io/projected/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-kube-api-access-7tvgs\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.812485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.812466 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.913877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.913791 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.914042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.913905 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvgs\" (UniqueName: \"kubernetes.io/projected/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-kube-api-access-7tvgs\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.914114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.914097 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.914691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.914662 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.916520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.916495 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.926896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.926867 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvgs\" (UniqueName: \"kubernetes.io/projected/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-kube-api-access-7tvgs\") pod \"maas-api-revoke-sub-5059f5ea-7nzhr\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:32.984187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:32.984148 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:33.272716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:33.272682 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr"] Oct 31 13:34:34.110658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:34.109802 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" event={"ID":"aa7ae030-1a22-4ecf-b1a4-070435bff7b8","Type":"ContainerStarted","Data":"53e39f755a90ad9f0c990cbf7090a9c4a5ca4d00a93118299272f3a70b5c094f"} Oct 31 13:34:34.110658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:34.109901 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" event={"ID":"aa7ae030-1a22-4ecf-b1a4-070435bff7b8","Type":"ContainerStarted","Data":"ad3030fb8f8ba7d46a5133dde53349e0c34a372d551c1bf312e03d9dc2010a3d"} Oct 31 13:34:34.169669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:34.169347 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" podStartSLOduration=2.169321384 podStartE2EDuration="2.169321384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:34.14510473 +0000 UTC m=+3768.322822714" watchObservedRunningTime="2026-10-31 13:34:34.169321384 +0000 UTC m=+3768.347039371" Oct 31 13:34:35.121947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:35.121911 5074 generic.go:356] "Generic (PLEG): container finished" podID="aa7ae030-1a22-4ecf-b1a4-070435bff7b8" containerID="53e39f755a90ad9f0c990cbf7090a9c4a5ca4d00a93118299272f3a70b5c094f" exitCode=0 Oct 31 13:34:35.122388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:35.121991 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" event={"ID":"aa7ae030-1a22-4ecf-b1a4-070435bff7b8","Type":"ContainerDied","Data":"53e39f755a90ad9f0c990cbf7090a9c4a5ca4d00a93118299272f3a70b5c094f"} Oct 31 13:34:36.326848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.326815 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:36.380131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.380096 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-kube-api-access-7tvgs\" (UniqueName: \"kubernetes.io/projected/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-kube-api-access-7tvgs\") pod \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " Oct 31 13:34:36.380354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.380220 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-xks-ca\") pod \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " Oct 31 13:34:36.380354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.380328 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-openshift-ca\") pod \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\" (UID: \"aa7ae030-1a22-4ecf-b1a4-070435bff7b8\") " Oct 31 13:34:36.380772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.380735 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-openshift-ca" pod "aa7ae030-1a22-4ecf-b1a4-070435bff7b8" (UID: "aa7ae030-1a22-4ecf-b1a4-070435bff7b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:34:36.381142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.381112 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:36.382549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.382515 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-kube-api-access-7tvgs" pod "aa7ae030-1a22-4ecf-b1a4-070435bff7b8" (UID: "aa7ae030-1a22-4ecf-b1a4-070435bff7b8"). InnerVolumeSpecName "kube-api-access-7tvgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:34:36.382733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.382571 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-xks-ca" pod "aa7ae030-1a22-4ecf-b1a4-070435bff7b8" (UID: "aa7ae030-1a22-4ecf-b1a4-070435bff7b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:34:36.511659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.507748 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvgs\" (UniqueName: \"kubernetes.io/projected/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-kube-api-access-7tvgs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:36.511659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:36.507798 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7ae030-1a22-4ecf-b1a4-070435bff7b8-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:37.141801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:37.141773 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" Oct 31 13:34:37.142002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:37.141810 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr" event={"ID":"aa7ae030-1a22-4ecf-b1a4-070435bff7b8","Type":"ContainerDied","Data":"ad3030fb8f8ba7d46a5133dde53349e0c34a372d551c1bf312e03d9dc2010a3d"} Oct 31 13:34:37.142002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:37.141860 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad3030fb8f8ba7d46a5133dde53349e0c34a372d551c1bf312e03d9dc2010a3d" Oct 31 13:34:40.861796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:40.861753 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:34:43.415820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.415784 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4"] Oct 31 13:34:43.416554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.416535 5074 cpu_manager.go:414] "containerMap: removing container" podUID="aa7ae030-1a22-4ecf-b1a4-070435bff7b8" containerName="revoke-keys" Oct 31 13:34:43.416627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.416556 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7ae030-1a22-4ecf-b1a4-070435bff7b8" containerName="revoke-keys" Oct 31 13:34:43.416741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.416729 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8ad859-e258-4a60-b804-ab05271038b5" containerName="istio-proxy" Oct 31 13:34:43.416782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.416750 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7ae030-1a22-4ecf-b1a4-070435bff7b8" containerName="revoke-keys" Oct 31 13:34:43.421203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.421177 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.427909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.427881 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:34:43.429221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.429169 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:34:43.435599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.435573 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4"] Oct 31 13:34:43.553298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.553259 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7nv\" (UniqueName: \"kubernetes.io/projected/87d401ef-4486-4a34-b614-8e3278fec2bc-kube-api-access-nb7nv\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.553566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.553534 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.553721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.553698 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.657092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.657044 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7nv\" (UniqueName: \"kubernetes.io/projected/87d401ef-4486-4a34-b614-8e3278fec2bc-kube-api-access-nb7nv\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.671706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.657530 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.672061 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.672031 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.672871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.669922 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.693655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.684986 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.698473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.698434 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7nv\" (UniqueName: \"kubernetes.io/projected/87d401ef-4486-4a34-b614-8e3278fec2bc-kube-api-access-nb7nv\") pod \"maas-api-revoke-sub-7ff23c11-jrvt4\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:43.744445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:43.744405 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:44.039177 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:34:44.039139 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d401ef_4486_4a34_b614_8e3278fec2bc.slice/crio-cc921e00d29ed9d3d54396ff1a3073f5f8f2c2865f29f020987b82028d23757d WatchSource:0}: Error finding container cc921e00d29ed9d3d54396ff1a3073f5f8f2c2865f29f020987b82028d23757d: Status 404 returned error can't find the container with id cc921e00d29ed9d3d54396ff1a3073f5f8f2c2865f29f020987b82028d23757d Oct 31 13:34:44.039336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:44.039239 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4"] Oct 31 13:34:44.270182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:44.270143 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" event={"ID":"87d401ef-4486-4a34-b614-8e3278fec2bc","Type":"ContainerStarted","Data":"cc921e00d29ed9d3d54396ff1a3073f5f8f2c2865f29f020987b82028d23757d"} Oct 31 13:34:45.283339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:45.283296 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" event={"ID":"87d401ef-4486-4a34-b614-8e3278fec2bc","Type":"ContainerStarted","Data":"26c15e593f6b5df00d9fd2c926709f7748acfb9ff10f0103530ea74a7d990a71"} Oct 31 13:34:45.303880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:45.302571 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" podStartSLOduration=2.30254495 podStartE2EDuration="2.30254495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:45.301895781 +0000 UTC m=+3779.479613771" watchObservedRunningTime="2026-10-31 13:34:45.30254495 +0000 UTC m=+3779.480262935" Oct 31 13:34:46.294160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:46.294125 5074 generic.go:356] "Generic (PLEG): container finished" podID="87d401ef-4486-4a34-b614-8e3278fec2bc" containerID="26c15e593f6b5df00d9fd2c926709f7748acfb9ff10f0103530ea74a7d990a71" exitCode=0 Oct 31 13:34:46.294747 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:46.294215 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" event={"ID":"87d401ef-4486-4a34-b614-8e3278fec2bc","Type":"ContainerDied","Data":"26c15e593f6b5df00d9fd2c926709f7748acfb9ff10f0103530ea74a7d990a71"} Oct 31 13:34:47.569289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.569220 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:47.654162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.654123 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-openshift-ca\") pod \"87d401ef-4486-4a34-b614-8e3278fec2bc\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " Oct 31 13:34:47.654376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.654296 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-xks-ca\") pod \"87d401ef-4486-4a34-b614-8e3278fec2bc\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " Oct 31 13:34:47.654376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.654368 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d401ef-4486-4a34-b614-8e3278fec2bc-kube-api-access-nb7nv\" (UniqueName: \"kubernetes.io/projected/87d401ef-4486-4a34-b614-8e3278fec2bc-kube-api-access-nb7nv\") pod \"87d401ef-4486-4a34-b614-8e3278fec2bc\" (UID: \"87d401ef-4486-4a34-b614-8e3278fec2bc\") " Oct 31 13:34:47.654550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.654518 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-openshift-ca" pod "87d401ef-4486-4a34-b614-8e3278fec2bc" (UID: "87d401ef-4486-4a34-b614-8e3278fec2bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:34:47.654935 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.654914 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:47.657035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.657001 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-xks-ca" pod "87d401ef-4486-4a34-b614-8e3278fec2bc" (UID: "87d401ef-4486-4a34-b614-8e3278fec2bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:34:47.657352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.657324 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d401ef-4486-4a34-b614-8e3278fec2bc-kube-api-access-nb7nv" pod "87d401ef-4486-4a34-b614-8e3278fec2bc" (UID: "87d401ef-4486-4a34-b614-8e3278fec2bc"). InnerVolumeSpecName "kube-api-access-nb7nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:34:47.756706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.756665 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d401ef-4486-4a34-b614-8e3278fec2bc-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:47.756706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:47.756702 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7nv\" (UniqueName: \"kubernetes.io/projected/87d401ef-4486-4a34-b614-8e3278fec2bc-kube-api-access-nb7nv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:48.314507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:48.314466 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" event={"ID":"87d401ef-4486-4a34-b614-8e3278fec2bc","Type":"ContainerDied","Data":"cc921e00d29ed9d3d54396ff1a3073f5f8f2c2865f29f020987b82028d23757d"} Oct 31 13:34:48.314507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:48.314504 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4" Oct 31 13:34:48.314838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:48.314530 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc921e00d29ed9d3d54396ff1a3073f5f8f2c2865f29f020987b82028d23757d" Oct 31 13:34:54.430304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.430267 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd"] Oct 31 13:34:54.431127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.431106 5074 cpu_manager.go:414] "containerMap: removing container" podUID="87d401ef-4486-4a34-b614-8e3278fec2bc" containerName="revoke-keys" Oct 31 13:34:54.431198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.431129 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d401ef-4486-4a34-b614-8e3278fec2bc" containerName="revoke-keys" Oct 31 13:34:54.431264 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.431255 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d401ef-4486-4a34-b614-8e3278fec2bc" containerName="revoke-keys" Oct 31 13:34:54.435284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.435264 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.444022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.443984 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:34:54.444338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.444312 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:34:54.444587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.444558 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd"] Oct 31 13:34:54.526885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.526845 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.527137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.527111 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.527273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.527252 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64dg\" (UniqueName: \"kubernetes.io/projected/1f5be180-554a-48d5-ba44-318e98170c30-kube-api-access-z64dg\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.629017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.628977 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.629211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.629091 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64dg\" (UniqueName: \"kubernetes.io/projected/1f5be180-554a-48d5-ba44-318e98170c30-kube-api-access-z64dg\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.629211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.629198 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.629886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.629858 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.631724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.631699 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.745656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.736261 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64dg\" (UniqueName: \"kubernetes.io/projected/1f5be180-554a-48d5-ba44-318e98170c30-kube-api-access-z64dg\") pod \"maas-api-revoke-sub-96f87e81-rwsxd\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:54.752740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:54.752703 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:55.044995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:55.044962 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd"] Oct 31 13:34:55.046606 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:34:55.046575 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5be180_554a_48d5_ba44_318e98170c30.slice/crio-2b53852e5728a2fa13c77cc4c3b19790438d8bf6d154884f0ec63b715666a5bc WatchSource:0}: Error finding container 2b53852e5728a2fa13c77cc4c3b19790438d8bf6d154884f0ec63b715666a5bc: Status 404 returned error can't find the container with id 2b53852e5728a2fa13c77cc4c3b19790438d8bf6d154884f0ec63b715666a5bc Oct 31 13:34:55.376245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:55.376154 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" event={"ID":"1f5be180-554a-48d5-ba44-318e98170c30","Type":"ContainerStarted","Data":"2b53852e5728a2fa13c77cc4c3b19790438d8bf6d154884f0ec63b715666a5bc"} Oct 31 13:34:55.565510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:55.565467 5074 scope.go:122] "RemoveContainer" containerID="bf8efa5430fdf8e5c716e1229e131fdc46d678b8c8acea3d54e5c31361d0fa3d" Oct 31 13:34:55.579542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:55.579514 5074 scope.go:122] "RemoveContainer" containerID="eb1c6ab10ca369fd30392e1351efbfbdc425aa5fd758a6a0d0bd253bcc120726" Oct 31 13:34:55.592107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:55.592085 5074 scope.go:122] "RemoveContainer" containerID="3abc7e8f26b38379c2df80c88afdfeba4fdf546e05732b9ac89de57f2a66bcc0" Oct 31 13:34:55.604587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:55.604561 5074 scope.go:122] "RemoveContainer" containerID="fbab7917d330bb125508d5ea27990042a328ae03eba52b20cd942d9ce51169ef" Oct 31 13:34:55.990373 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:34:55.990334 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:34:56.388055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:56.387961 5074 generic.go:356] "Generic (PLEG): container finished" podID="1f5be180-554a-48d5-ba44-318e98170c30" containerID="87fd37db3c4a3d0274eff16d6d46b8e00cb937932f109fb6d7bc8f98ab9bfad9" exitCode=0 Oct 31 13:34:56.388263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:56.388243 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" event={"ID":"1f5be180-554a-48d5-ba44-318e98170c30","Type":"ContainerDied","Data":"87fd37db3c4a3d0274eff16d6d46b8e00cb937932f109fb6d7bc8f98ab9bfad9"} Oct 31 13:34:57.602007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.601973 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:57.725418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.725377 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-openshift-ca\") pod \"1f5be180-554a-48d5-ba44-318e98170c30\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " Oct 31 13:34:57.725602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.725445 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-xks-ca\") pod \"1f5be180-554a-48d5-ba44-318e98170c30\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " Oct 31 13:34:57.725602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.725512 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5be180-554a-48d5-ba44-318e98170c30-kube-api-access-z64dg\" (UniqueName: \"kubernetes.io/projected/1f5be180-554a-48d5-ba44-318e98170c30-kube-api-access-z64dg\") pod \"1f5be180-554a-48d5-ba44-318e98170c30\" (UID: \"1f5be180-554a-48d5-ba44-318e98170c30\") " Oct 31 13:34:57.725806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.725780 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-openshift-ca" pod "1f5be180-554a-48d5-ba44-318e98170c30" (UID: "1f5be180-554a-48d5-ba44-318e98170c30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:34:57.726408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.726385 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:57.727705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.727677 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5be180-554a-48d5-ba44-318e98170c30-kube-api-access-z64dg" pod "1f5be180-554a-48d5-ba44-318e98170c30" (UID: "1f5be180-554a-48d5-ba44-318e98170c30"). InnerVolumeSpecName "kube-api-access-z64dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:34:57.727821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.727715 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-xks-ca" pod "1f5be180-554a-48d5-ba44-318e98170c30" (UID: "1f5be180-554a-48d5-ba44-318e98170c30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:34:57.827922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.827878 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5be180-554a-48d5-ba44-318e98170c30-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:57.827922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:57.827926 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64dg\" (UniqueName: \"kubernetes.io/projected/1f5be180-554a-48d5-ba44-318e98170c30-kube-api-access-z64dg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:58.410405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:58.410354 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" Oct 31 13:34:58.410405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:58.410383 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd" event={"ID":"1f5be180-554a-48d5-ba44-318e98170c30","Type":"ContainerDied","Data":"2b53852e5728a2fa13c77cc4c3b19790438d8bf6d154884f0ec63b715666a5bc"} Oct 31 13:34:58.410740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:34:58.410441 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b53852e5728a2fa13c77cc4c3b19790438d8bf6d154884f0ec63b715666a5bc" Oct 31 13:35:02.028355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:02.028308 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg"] Oct 31 13:35:02.032616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:02.032585 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92074962-sknrg"] Oct 31 13:35:03.942117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:03.942075 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296b3798-ffd7-4ca7-b409-b966928828f7" path="/var/lib/kubelet/pods/296b3798-ffd7-4ca7-b409-b966928828f7/volumes" Oct 31 13:35:26.040905 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:35:26.040820 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:35:26.773821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:26.773781 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:35:26.774163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:26.774121 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:35:26.849484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:26.849440 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np"] Oct 31 13:35:26.849756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:26.849719 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" podUID="77c28c8e-c760-4e94-9530-a4cc7ea2d908" containerName="maas-api" containerID="cri-o://f920d8a7e08e2377184418560edceeec7830cf3b3bb10beddbf60855ef18dbe2" gracePeriod=30 Oct 31 13:35:27.249809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.249746 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" podUID="77c28c8e-c760-4e94-9530-a4cc7ea2d908" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.44:8443/health\": dial tcp 10.132.1.44:8443: connect: connection refused" Oct 31 13:35:27.421654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.418835 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:35:27.421654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.418920 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:35:27.953933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.953899 5074 generic.go:356] "Generic (PLEG): container finished" podID="77c28c8e-c760-4e94-9530-a4cc7ea2d908" containerID="f920d8a7e08e2377184418560edceeec7830cf3b3bb10beddbf60855ef18dbe2" exitCode=0 Oct 31 13:35:27.954754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.954723 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" event={"ID":"77c28c8e-c760-4e94-9530-a4cc7ea2d908","Type":"ContainerDied","Data":"f920d8a7e08e2377184418560edceeec7830cf3b3bb10beddbf60855ef18dbe2"} Oct 31 13:35:27.954984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.954777 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" event={"ID":"77c28c8e-c760-4e94-9530-a4cc7ea2d908","Type":"ContainerDied","Data":"dadd24ef8b50db5c334d3aacfcf97e063a4f01540b910928fdb07b0c3c998161"} Oct 31 13:35:27.954984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.954797 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dadd24ef8b50db5c334d3aacfcf97e063a4f01540b910928fdb07b0c3c998161" Oct 31 13:35:27.955128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:27.955076 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:35:28.140723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.121875 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c28c8e-c760-4e94-9530-a4cc7ea2d908-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c28c8e-c760-4e94-9530-a4cc7ea2d908-openshift-service-ca\") pod \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " Oct 31 13:35:28.140723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.122279 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-metrics-certs\") pod \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " Oct 31 13:35:28.140723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.122598 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c28c8e-c760-4e94-9530-a4cc7ea2d908-kube-api-access-pkkdp\" (UniqueName: \"kubernetes.io/projected/77c28c8e-c760-4e94-9530-a4cc7ea2d908-kube-api-access-pkkdp\") pod \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " Oct 31 13:35:28.140723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.122724 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-maas-api-tls\") pod \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\" (UID: \"77c28c8e-c760-4e94-9530-a4cc7ea2d908\") " Oct 31 13:35:28.166003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.163173 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c28c8e-c760-4e94-9530-a4cc7ea2d908-openshift-service-ca" pod "77c28c8e-c760-4e94-9530-a4cc7ea2d908" (UID: "77c28c8e-c760-4e94-9530-a4cc7ea2d908"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:28.166003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.164557 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-metrics-certs" pod "77c28c8e-c760-4e94-9530-a4cc7ea2d908" (UID: "77c28c8e-c760-4e94-9530-a4cc7ea2d908"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:35:28.184444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.184405 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-maas-api-tls" pod "77c28c8e-c760-4e94-9530-a4cc7ea2d908" (UID: "77c28c8e-c760-4e94-9530-a4cc7ea2d908"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:35:28.186318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.186074 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c28c8e-c760-4e94-9530-a4cc7ea2d908-kube-api-access-pkkdp" pod "77c28c8e-c760-4e94-9530-a4cc7ea2d908" (UID: "77c28c8e-c760-4e94-9530-a4cc7ea2d908"). InnerVolumeSpecName "kube-api-access-pkkdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:28.226248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.226215 5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-metrics-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:28.226248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.226246 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkkdp\" (UniqueName: \"kubernetes.io/projected/77c28c8e-c760-4e94-9530-a4cc7ea2d908-kube-api-access-pkkdp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:28.226446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.226262 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77c28c8e-c760-4e94-9530-a4cc7ea2d908-maas-api-tls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:28.226446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.226287 5074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c28c8e-c760-4e94-9530-a4cc7ea2d908-openshift-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:28.967097 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:28.967050 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np" Oct 31 13:35:29.019336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:29.019300 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np"] Oct 31 13:35:29.020486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:29.020462 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a39960-666c9874dd-zv6np"] Oct 31 13:35:29.966124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:29.966087 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c28c8e-c760-4e94-9530-a4cc7ea2d908" path="/var/lib/kubelet/pods/77c28c8e-c760-4e94-9530-a4cc7ea2d908/volumes" Oct 31 13:35:30.773283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:30.773246 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx"] Oct 31 13:35:30.773810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:30.773457 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" containerID="cri-o://8a5df8eb03d44a6940bc33be6aff01e996d7e6c44d9c667f04bd97a508078e69" gracePeriod=30 Oct 31 13:35:35.218659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:35.218606 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm"] Oct 31 13:35:35.891493 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:35:35.891445 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda453269b_dc26_4e66_833f_a84a512e0c10.slice/crio-8a5df8eb03d44a6940bc33be6aff01e996d7e6c44d9c667f04bd97a508078e69.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda453269b_dc26_4e66_833f_a84a512e0c10.slice/crio-conmon-8a5df8eb03d44a6940bc33be6aff01e996d7e6c44d9c667f04bd97a508078e69.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:35:36.041963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.041929 5074 generic.go:356] "Generic (PLEG): container finished" podID="a453269b-dc26-4e66-833f-a84a512e0c10" containerID="8a5df8eb03d44a6940bc33be6aff01e996d7e6c44d9c667f04bd97a508078e69" exitCode=0 Oct 31 13:35:36.042179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.042054 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" event={"ID":"a453269b-dc26-4e66-833f-a84a512e0c10","Type":"ContainerDied","Data":"8a5df8eb03d44a6940bc33be6aff01e996d7e6c44d9c667f04bd97a508078e69"} Oct 31 13:35:36.096256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.096226 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:35:36.128584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.128082 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-credential-socket\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.128584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.128245 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-envoy\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.128584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.128304 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-certs\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.128584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.128571 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-credential-socket" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:36.128983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.128958 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-data\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.129064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.129047 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a453269b-dc26-4e66-833f-a84a512e0c10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a453269b-dc26-4e66-833f-a84a512e0c10-istio-podinfo\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.129087 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-istio-token\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-istio-token\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.129146 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-kube-api-access-rf5dh\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-kube-api-access-rf5dh\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.129223 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a453269b-dc26-4e66-833f-a84a512e0c10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a453269b-dc26-4e66-833f-a84a512e0c10-istiod-ca-cert\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.129264 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-socket\") pod \"a453269b-dc26-4e66-833f-a84a512e0c10\" (UID: \"a453269b-dc26-4e66-833f-a84a512e0c10\") " Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.130119 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.130330 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-socket" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.131140 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a453269b-dc26-4e66-833f-a84a512e0c10-istiod-ca-cert" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:36.131226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.131202 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-data" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:36.131814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.131284 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-certs" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:36.133431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.133395 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-envoy" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:36.133914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.133885 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a453269b-dc26-4e66-833f-a84a512e0c10-istio-podinfo" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:35:36.135517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.135484 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-istio-token" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:36.138761 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.138738 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-kube-api-access-rf5dh" pod "a453269b-dc26-4e66-833f-a84a512e0c10" (UID: "a453269b-dc26-4e66-833f-a84a512e0c10"). InnerVolumeSpecName "kube-api-access-rf5dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:36.231529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231492 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.231529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231528 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.231970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231538 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.231970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231548 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a453269b-dc26-4e66-833f-a84a512e0c10-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.231970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231556 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.231970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231565 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5dh\" (UniqueName: \"kubernetes.io/projected/a453269b-dc26-4e66-833f-a84a512e0c10-kube-api-access-rf5dh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.231970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231576 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a453269b-dc26-4e66-833f-a84a512e0c10-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:36.231970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:36.231590 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a453269b-dc26-4e66-833f-a84a512e0c10-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:37.094553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.091832 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c"] Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.142599 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.142676 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.142739 5074 cpu_manager.go:414] "containerMap: removing container" podUID="77c28c8e-c760-4e94-9530-a4cc7ea2d908" containerName="maas-api" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.142749 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c28c8e-c760-4e94-9530-a4cc7ea2d908" containerName="maas-api" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.142785 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1f5be180-554a-48d5-ba44-318e98170c30" containerName="revoke-keys" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.142793 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5be180-554a-48d5-ba44-318e98170c30" containerName="revoke-keys" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.143011 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" containerName="istio-proxy" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.143033 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c28c8e-c760-4e94-9530-a4cc7ea2d908" containerName="maas-api" Oct 31 13:35:37.144683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.143049 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5be180-554a-48d5-ba44-318e98170c30" containerName="revoke-keys" Oct 31 13:35:37.149405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.148567 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.166027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.165973 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" event={"ID":"a453269b-dc26-4e66-833f-a84a512e0c10","Type":"ContainerDied","Data":"55e0a657f5e9302840b86970c457bd1f18f8b62a1db4086151bd7b9639f18ee2"} Oct 31 13:35:37.166197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.166068 5074 scope.go:122] "RemoveContainer" containerID="8a5df8eb03d44a6940bc33be6aff01e996d7e6c44d9c667f04bd97a508078e69" Oct 31 13:35:37.166260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.166234 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx" Oct 31 13:35:37.171487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.171451 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c"] Oct 31 13:35:37.225862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.224532 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx"] Oct 31 13:35:37.230102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.230075 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-ls8fx"] Oct 31 13:35:37.256995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.256967 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257091 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257291 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257342 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257430 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257470 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257528 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257578 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.257647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.257612 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpnbv\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-kube-api-access-gpnbv\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.358736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.358618 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.358893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.358747 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.358950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.358924 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.358953 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.358989 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359032 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359074 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359134 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359192 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpnbv\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-kube-api-access-gpnbv\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359334 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359440 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359460 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359481 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.359736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.359718 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.361352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.361319 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.361475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.361379 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.373933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.373907 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.374168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.374148 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpnbv\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-kube-api-access-gpnbv\") pod \"maas-default-gateway-openshift-default-fc8c56547-2r62c\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.498426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.498391 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:37.796337 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:35:37.796301 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845e6ed7_6c57_445f_acc3_8bd3f71bb72f.slice/crio-95050176605ed35bf1b78eebdfdc3624e648e280382b1b6d1b0b3b676c37fd3a WatchSource:0}: Error finding container 95050176605ed35bf1b78eebdfdc3624e648e280382b1b6d1b0b3b676c37fd3a: Status 404 returned error can't find the container with id 95050176605ed35bf1b78eebdfdc3624e648e280382b1b6d1b0b3b676c37fd3a Oct 31 13:35:37.798693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.798661 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c"] Oct 31 13:35:37.800309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.800266 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:37.800410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.800358 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:37.800410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.800404 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:37.943714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:37.943677 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a453269b-dc26-4e66-833f-a84a512e0c10" path="/var/lib/kubelet/pods/a453269b-dc26-4e66-833f-a84a512e0c10/volumes" Oct 31 13:35:38.178708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:38.178558 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" event={"ID":"845e6ed7-6c57-445f-acc3-8bd3f71bb72f","Type":"ContainerStarted","Data":"a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e"} Oct 31 13:35:38.178708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:38.178674 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" event={"ID":"845e6ed7-6c57-445f-acc3-8bd3f71bb72f","Type":"ContainerStarted","Data":"95050176605ed35bf1b78eebdfdc3624e648e280382b1b6d1b0b3b676c37fd3a"} Oct 31 13:35:38.216284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:38.216221 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podStartSLOduration=1.2162017139999999 podStartE2EDuration="1.216201714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:38.211022049 +0000 UTC m=+3832.388740040" watchObservedRunningTime="2026-10-31 13:35:38.216201714 +0000 UTC m=+3832.393919714" Oct 31 13:35:38.499447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:38.499406 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:38.501418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:38.501386 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:38.501566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:38.501459 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:39.500219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:39.500166 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:39.500898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:39.500239 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:40.220486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:40.220432 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" podUID="54fcd3d7-75a1-4486-bb38-e102922aaada" containerName="payload-pre-processing" containerID="cri-o://2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975" gracePeriod=25 Oct 31 13:35:40.499605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:40.499028 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:40.499605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:40.499114 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:41.500172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:41.500131 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:41.500835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:41.500783 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:42.499076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:42.499033 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:42.499418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:42.499379 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:43.499147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:43.499099 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:43.499720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:43.499190 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:44.343930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:44.343892 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" podUID="54fcd3d7-75a1-4486-bb38-e102922aaada" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:35:44.508667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:44.508573 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:44.509217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:44.508676 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:45.502151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:45.502107 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-2r62c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Oct 31 13:35:45.502467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:45.502433 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Oct 31 13:35:46.971843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:46.971805 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:46.972935 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:46.972903 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:46.974540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:46.974510 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:35:54.343784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:54.343747 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" podUID="54fcd3d7-75a1-4486-bb38-e102922aaada" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:35:55.651227 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:35:55.651192 5074 scope.go:122] "RemoveContainer" containerID="f920d8a7e08e2377184418560edceeec7830cf3b3bb10beddbf60855ef18dbe2" Oct 31 13:35:55.987666 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:35:55.987598 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:36:00.252905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.252877 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99"] Oct 31 13:36:00.253273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.253088 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" podUID="02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" containerName="discovery" containerID="cri-o://733840636f3817b17b96f3f0ef0f72d72f75a262f1bf9fb05268e5601f508997" gracePeriod=30 Oct 31 13:36:00.491019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.490257 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:36:00.491767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.491738 5074 generic.go:356] "Generic (PLEG): container finished" podID="54fcd3d7-75a1-4486-bb38-e102922aaada" containerID="2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975" exitCode=0 Oct 31 13:36:00.491893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.491851 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" event={"ID":"54fcd3d7-75a1-4486-bb38-e102922aaada","Type":"ContainerDied","Data":"2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975"} Oct 31 13:36:00.491958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.491901 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" event={"ID":"54fcd3d7-75a1-4486-bb38-e102922aaada","Type":"ContainerDied","Data":"1c8562e07049f50da8389aa422549544641c86f90bf55425470a243f8647ba74"} Oct 31 13:36:00.491958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.491931 5074 scope.go:122] "RemoveContainer" containerID="2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975" Oct 31 13:36:00.495033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.494564 5074 generic.go:356] "Generic (PLEG): container finished" podID="02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" containerID="733840636f3817b17b96f3f0ef0f72d72f75a262f1bf9fb05268e5601f508997" exitCode=0 Oct 31 13:36:00.495033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.494961 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" event={"ID":"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f","Type":"ContainerDied","Data":"733840636f3817b17b96f3f0ef0f72d72f75a262f1bf9fb05268e5601f508997"} Oct 31 13:36:00.510130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.510077 5074 scope.go:122] "RemoveContainer" containerID="2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975" Oct 31 13:36:00.511211 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:36:00.511076 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975\": container with ID starting with 2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975 not found: ID does not exist" containerID="2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975" Oct 31 13:36:00.511211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.511135 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975"} err="failed to get container status \"2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975\": rpc error: code = NotFound desc = could not find container \"2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975\": container with ID starting with 2059c382a617af890fd9baf6fdbc786c0717d443591e5be306c4c717542a0975 not found: ID does not exist" Oct 31 13:36:00.526010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.525303 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fcd3d7-75a1-4486-bb38-e102922aaada-kube-api-access-cmtpp\" (UniqueName: \"kubernetes.io/projected/54fcd3d7-75a1-4486-bb38-e102922aaada-kube-api-access-cmtpp\") pod \"54fcd3d7-75a1-4486-bb38-e102922aaada\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " Oct 31 13:36:00.526010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.525672 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fcd3d7-75a1-4486-bb38-e102922aaada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fcd3d7-75a1-4486-bb38-e102922aaada-plugins-config-volume\") pod \"54fcd3d7-75a1-4486-bb38-e102922aaada\" (UID: \"54fcd3d7-75a1-4486-bb38-e102922aaada\") " Oct 31 13:36:00.537207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.537158 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fcd3d7-75a1-4486-bb38-e102922aaada-kube-api-access-cmtpp" pod "54fcd3d7-75a1-4486-bb38-e102922aaada" (UID: "54fcd3d7-75a1-4486-bb38-e102922aaada"). InnerVolumeSpecName "kube-api-access-cmtpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:00.539328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.539292 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fcd3d7-75a1-4486-bb38-e102922aaada-plugins-config-volume" pod "54fcd3d7-75a1-4486-bb38-e102922aaada" (UID: "54fcd3d7-75a1-4486-bb38-e102922aaada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:00.629133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.628987 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fcd3d7-75a1-4486-bb38-e102922aaada-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.629133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.629026 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtpp\" (UniqueName: \"kubernetes.io/projected/54fcd3d7-75a1-4486-bb38-e102922aaada-kube-api-access-cmtpp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.654284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.654258 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.729537 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-cacerts\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-cacerts\") pod \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.729603 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-kubeconfig\") pod \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.729747 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-ca-configmap\") pod \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.729882 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-kube-api-access-xfhxj\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-kube-api-access-xfhxj\") pod \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.729979 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-local-certs\") pod \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.730094 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-token\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-token\") pod \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.730228 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-dns-cert\") pod \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\" (UID: \"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f\") " Oct 31 13:36:00.731025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.730710 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-ca-configmap" pod "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" (UID: "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:00.731697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.731107 5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-ca-configmap\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.732724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.732695 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-cacerts" pod "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" (UID: "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:00.734508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.734452 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-kube-api-access-xfhxj" pod "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" (UID: "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f"). InnerVolumeSpecName "kube-api-access-xfhxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:00.735027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.734997 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-token" pod "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" (UID: "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:00.735137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.735098 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-dns-cert" pod "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" (UID: "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:00.735787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.735765 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-local-certs" pod "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" (UID: "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:00.736893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.736875 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-kubeconfig" pod "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" (UID: "02e0d99c-c28b-4b5c-be0e-d9e47fdf522f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:00.832858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.832823 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhxj\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-kube-api-access-xfhxj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.832858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.832854 5074 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-local-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.832858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.832865 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.833180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.832877 5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-csr-dns-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.833180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.832885 5074 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-cacerts\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.833180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:00.832894 5074 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f-istio-kubeconfig\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:01.504689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.504561 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm" Oct 31 13:36:01.506568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.506534 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" Oct 31 13:36:01.506568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.506546 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99" event={"ID":"02e0d99c-c28b-4b5c-be0e-d9e47fdf522f","Type":"ContainerDied","Data":"7e9b203369a6d5500f29a74f147c00be2c4eb4912587ab1ee45a005369eb2ff7"} Oct 31 13:36:01.506839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.506674 5074 scope.go:122] "RemoveContainer" containerID="733840636f3817b17b96f3f0ef0f72d72f75a262f1bf9fb05268e5601f508997" Oct 31 13:36:01.537750 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.537721 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm"] Oct 31 13:36:01.542344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.542322 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a13836-c5c4db4d7-mj4hm"] Oct 31 13:36:01.555100 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.555067 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99"] Oct 31 13:36:01.561056 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.561032 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8pb99"] Oct 31 13:36:01.693836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.693802 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl"] Oct 31 13:36:01.694312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:01.694278 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" podUID="015076d4-f1f7-47af-94fb-09395734c846" containerName="maas-api" containerID="cri-o://a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb" gracePeriod=30 Oct 31 13:36:02.173706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.161846 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" path="/var/lib/kubelet/pods/02e0d99c-c28b-4b5c-be0e-d9e47fdf522f/volumes" Oct 31 13:36:02.232660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.222918 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fcd3d7-75a1-4486-bb38-e102922aaada" path="/var/lib/kubelet/pods/54fcd3d7-75a1-4486-bb38-e102922aaada/volumes" Oct 31 13:36:02.460098 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.458760 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:36:02.519619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.519578 5074 generic.go:356] "Generic (PLEG): container finished" podID="015076d4-f1f7-47af-94fb-09395734c846" containerID="a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb" exitCode=0 Oct 31 13:36:02.520205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.519673 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" Oct 31 13:36:02.520205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.519756 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" event={"ID":"015076d4-f1f7-47af-94fb-09395734c846","Type":"ContainerDied","Data":"a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb"} Oct 31 13:36:02.520205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.519812 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl" event={"ID":"015076d4-f1f7-47af-94fb-09395734c846","Type":"ContainerDied","Data":"a960f03caa81553a140b0e4a85d4d7a76e72e525d06ecff5121df6e3e996709d"} Oct 31 13:36:02.520205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.519844 5074 scope.go:122] "RemoveContainer" containerID="a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb" Oct 31 13:36:02.532338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.532311 5074 scope.go:122] "RemoveContainer" containerID="a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb" Oct 31 13:36:02.532723 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:36:02.532688 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb\": container with ID starting with a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb not found: ID does not exist" containerID="a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb" Oct 31 13:36:02.532843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.532732 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb"} err="failed to get container status \"a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb\": rpc error: code = NotFound desc = could not find container \"a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb\": container with ID starting with a43a0962eba09be0af1920cab520abc299ac5c5fd2a42a032e028168f1c20feb not found: ID does not exist" Oct 31 13:36:02.548652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.548608 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015076d4-f1f7-47af-94fb-09395734c846-kube-api-access-qwk8n\" (UniqueName: \"kubernetes.io/projected/015076d4-f1f7-47af-94fb-09395734c846-kube-api-access-qwk8n\") pod \"015076d4-f1f7-47af-94fb-09395734c846\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " Oct 31 13:36:02.548806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.548685 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-maas-api-tls\") pod \"015076d4-f1f7-47af-94fb-09395734c846\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " Oct 31 13:36:02.549010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.548988 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-metrics-certs\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-metrics-certs\") pod \"015076d4-f1f7-47af-94fb-09395734c846\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " Oct 31 13:36:02.549132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.549099 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015076d4-f1f7-47af-94fb-09395734c846-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015076d4-f1f7-47af-94fb-09395734c846-openshift-service-ca\") pod \"015076d4-f1f7-47af-94fb-09395734c846\" (UID: \"015076d4-f1f7-47af-94fb-09395734c846\") " Oct 31 13:36:02.549673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.549620 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015076d4-f1f7-47af-94fb-09395734c846-openshift-service-ca" pod "015076d4-f1f7-47af-94fb-09395734c846" (UID: "015076d4-f1f7-47af-94fb-09395734c846"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:02.549964 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.549935 5074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015076d4-f1f7-47af-94fb-09395734c846-openshift-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:02.551115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.551088 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-metrics-certs" pod "015076d4-f1f7-47af-94fb-09395734c846" (UID: "015076d4-f1f7-47af-94fb-09395734c846"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:02.551379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.551358 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015076d4-f1f7-47af-94fb-09395734c846-kube-api-access-qwk8n" pod "015076d4-f1f7-47af-94fb-09395734c846" (UID: "015076d4-f1f7-47af-94fb-09395734c846"). InnerVolumeSpecName "kube-api-access-qwk8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:02.552028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.552009 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-maas-api-tls" pod "015076d4-f1f7-47af-94fb-09395734c846" (UID: "015076d4-f1f7-47af-94fb-09395734c846"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:02.653130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.651873 5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-metrics-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:02.653130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.651916 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwk8n\" (UniqueName: \"kubernetes.io/projected/015076d4-f1f7-47af-94fb-09395734c846-kube-api-access-qwk8n\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:02.653130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.651933 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015076d4-f1f7-47af-94fb-09395734c846-maas-api-tls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:02.846427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.846396 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl"] Oct 31 13:36:02.852755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:02.852726 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a13836-75d856f76c-vg4xl"] Oct 31 13:36:03.941155 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:03.941118 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015076d4-f1f7-47af-94fb-09395734c846" path="/var/lib/kubelet/pods/015076d4-f1f7-47af-94fb-09395734c846/volumes" Oct 31 13:36:07.292487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:07.292448 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:36:10.403003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:10.402959 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx"] Oct 31 13:36:10.403533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:10.403252 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" containerID="cri-o://28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10" gracePeriod=30 Oct 31 13:36:12.029250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.029212 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs"] Oct 31 13:36:12.030973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.030940 5074 cpu_manager.go:414] "containerMap: removing container" podUID="015076d4-f1f7-47af-94fb-09395734c846" containerName="maas-api" Oct 31 13:36:12.030973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.030971 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015076d4-f1f7-47af-94fb-09395734c846" containerName="maas-api" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.031002 5074 cpu_manager.go:414] "containerMap: removing container" podUID="54fcd3d7-75a1-4486-bb38-e102922aaada" containerName="payload-pre-processing" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.031011 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fcd3d7-75a1-4486-bb38-e102922aaada" containerName="payload-pre-processing" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.031043 5074 cpu_manager.go:414] "containerMap: removing container" podUID="02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" containerName="discovery" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.031051 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" containerName="discovery" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.031211 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e0d99c-c28b-4b5c-be0e-d9e47fdf522f" containerName="discovery" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.031238 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="015076d4-f1f7-47af-94fb-09395734c846" containerName="maas-api" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.031249 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fcd3d7-75a1-4486-bb38-e102922aaada" containerName="payload-pre-processing" Oct 31 13:36:12.055504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.048924 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.058862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.056864 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e7896150-openshift-default-dockercfg-d6npt\"" Oct 31 13:36:12.071773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.071738 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs"] Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175162 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-socket\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175263 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-data\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175301 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-certs\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175406 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-credential-socket\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175473 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-podinfo\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175499 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2vp\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-kube-api-access-vm2vp\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175618 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c86405-b843-4803-8b7a-2698ce6dc7be-istiod-ca-cert\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175668 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-envoy\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.176008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.175697 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-token\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.280737 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-socket\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.280838 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-data\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.280874 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-certs\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.280965 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-credential-socket\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.281028 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-podinfo\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.281054 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2vp\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-kube-api-access-vm2vp\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.281160 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c86405-b843-4803-8b7a-2698ce6dc7be-istiod-ca-cert\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.281191 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-envoy\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.289793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.281217 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-token\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.326683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.307765 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c86405-b843-4803-8b7a-2698ce6dc7be-istiod-ca-cert\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.326683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.309950 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-certs\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.326683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.310207 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-data\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.326683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.310437 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-credential-socket\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.327647 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.327596 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-socket\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.405657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.394505 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-envoy\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.405657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.395085 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-podinfo\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.503918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.502102 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-token\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.546180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.529023 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2vp\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-kube-api-access-vm2vp\") pod \"e2e-mt-e7896150-openshift-default-744c455f69-jd2fs\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:12.743253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:12.743215 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:13.039002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.038973 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs"] Oct 31 13:36:13.042947 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:36:13.042920 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c86405_b843_4803_8b7a_2698ce6dc7be.slice/crio-16e050b38f0d119b9e33c24171bcc430b0c5c07d899df8915a6fbe1a7e89d31e WatchSource:0}: Error finding container 16e050b38f0d119b9e33c24171bcc430b0c5c07d899df8915a6fbe1a7e89d31e: Status 404 returned error can't find the container with id 16e050b38f0d119b9e33c24171bcc430b0c5c07d899df8915a6fbe1a7e89d31e Oct 31 13:36:13.046502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.046456 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:13.046602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.046552 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:13.046659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.046599 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:13.712933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.712889 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" event={"ID":"78c86405-b843-4803-8b7a-2698ce6dc7be","Type":"ContainerStarted","Data":"df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca"} Oct 31 13:36:13.713124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.712958 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" event={"ID":"78c86405-b843-4803-8b7a-2698ce6dc7be","Type":"ContainerStarted","Data":"16e050b38f0d119b9e33c24171bcc430b0c5c07d899df8915a6fbe1a7e89d31e"} Oct 31 13:36:13.743879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.743847 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:13.749342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:13.749296 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podStartSLOduration=2.7492825400000003 podStartE2EDuration="2.74928254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:13.742936858 +0000 UTC m=+3867.920654843" watchObservedRunningTime="2026-10-31 13:36:13.74928254 +0000 UTC m=+3867.927000602" Oct 31 13:36:14.749770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:14.749737 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:15.713854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.713782 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:36:15.736261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.736228 5074 generic.go:356] "Generic (PLEG): container finished" podID="8c639106-3f06-4fe9-9883-f578170b9015" containerID="28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10" exitCode=0 Oct 31 13:36:15.736387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.736273 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" Oct 31 13:36:15.736387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.736271 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" event={"ID":"8c639106-3f06-4fe9-9883-f578170b9015","Type":"ContainerDied","Data":"28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10"} Oct 31 13:36:15.736387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.736335 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx" event={"ID":"8c639106-3f06-4fe9-9883-f578170b9015","Type":"ContainerDied","Data":"945869b98129baca74ae632a767859b9174af5938c9c6d73b2c7fd32a4250d42"} Oct 31 13:36:15.736387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.736359 5074 scope.go:122] "RemoveContainer" containerID="28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10" Oct 31 13:36:15.736618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.736603 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:15.737619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.737603 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:36:15.749930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.749901 5074 scope.go:122] "RemoveContainer" containerID="9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e" Oct 31 13:36:15.761331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.761313 5074 scope.go:122] "RemoveContainer" containerID="28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10" Oct 31 13:36:15.766064 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:36:15.766038 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10\": container with ID starting with 28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10 not found: ID does not exist" containerID="28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10" Oct 31 13:36:15.766179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.766074 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10"} err="failed to get container status \"28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10\": rpc error: code = NotFound desc = could not find container \"28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10\": container with ID starting with 28611705f47abbbcefcb6d82ba44d4a0b34a8d3f7d18faa489a9e7eec5d9ab10 not found: ID does not exist" Oct 31 13:36:15.766179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.766108 5074 scope.go:122] "RemoveContainer" containerID="9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e" Oct 31 13:36:15.766435 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:36:15.766408 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e\": container with ID starting with 9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e not found: ID does not exist" containerID="9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e" Oct 31 13:36:15.766499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.766445 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e"} err="failed to get container status \"9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e\": rpc error: code = NotFound desc = could not find container \"9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e\": container with ID starting with 9f5ae12548b5bd18b170c52e3871be49d04a64413df772e67f69e6ec428e464e not found: ID does not exist" Oct 31 13:36:15.788400 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.788371 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-certs\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.789004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.788978 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-kube-api-access-ss4dk\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-kube-api-access-ss4dk\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.789210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.789191 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-envoy\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.789345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.788861 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-certs" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:15.789494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.789473 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c639106-3f06-4fe9-9883-f578170b9015-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c639106-3f06-4fe9-9883-f578170b9015-istiod-ca-cert\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.789686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.789666 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-socket\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.790254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.790236 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-istio-token\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-istio-token\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.790427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.790411 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-data\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.790787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.790767 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-credential-socket\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.791015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.790998 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c639106-3f06-4fe9-9883-f578170b9015-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c639106-3f06-4fe9-9883-f578170b9015-istio-podinfo\") pod \"8c639106-3f06-4fe9-9883-f578170b9015\" (UID: \"8c639106-3f06-4fe9-9883-f578170b9015\") " Oct 31 13:36:15.792157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.792131 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.793298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.792883 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c639106-3f06-4fe9-9883-f578170b9015-istiod-ca-cert" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:15.795172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.794479 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-credential-socket" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:15.795449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.795416 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-data" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:15.799951 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.798444 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-istio-token" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:15.817737 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.814982 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-kube-api-access-ss4dk" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "kube-api-access-ss4dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:15.819014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.818977 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-socket" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:15.823278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.823237 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c639106-3f06-4fe9-9883-f578170b9015-istio-podinfo" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:36:15.830178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.830149 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-envoy" pod "8c639106-3f06-4fe9-9883-f578170b9015" (UID: "8c639106-3f06-4fe9-9883-f578170b9015"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896441 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896482 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c639106-3f06-4fe9-9883-f578170b9015-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896499 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4dk\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-kube-api-access-ss4dk\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896518 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896532 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c639106-3f06-4fe9-9883-f578170b9015-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896548 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896562 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c639106-3f06-4fe9-9883-f578170b9015-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:15.897157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:15.896575 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c639106-3f06-4fe9-9883-f578170b9015-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.114818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:16.114783 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx"] Oct 31 13:36:16.153076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:16.153041 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a13836-openshift-default-5fcb95fb65-brvzx"] Oct 31 13:36:17.566939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.566903 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j"] Oct 31 13:36:17.567626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.567611 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" Oct 31 13:36:17.567695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.567627 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" Oct 31 13:36:17.567695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.567655 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" Oct 31 13:36:17.567695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.567661 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" Oct 31 13:36:17.567817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.567807 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" Oct 31 13:36:17.567868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.567820 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c639106-3f06-4fe9-9883-f578170b9015" containerName="istio-proxy" Oct 31 13:36:17.573172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.573151 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:17.580195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.580163 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e7896150\"" Oct 31 13:36:17.583035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.582882 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j"] Oct 31 13:36:17.632248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.632214 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwpl\" (UniqueName: \"kubernetes.io/projected/7af7844c-8037-47ed-8593-84a92852c243-kube-api-access-jjwpl\") pod \"payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:17.632463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.632387 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af7844c-8037-47ed-8593-84a92852c243-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:17.653897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.653863 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62"] Oct 31 13:36:17.658382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.658364 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:17.663176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.663103 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e7896150-dockercfg-jb64t\"" Oct 31 13:36:17.665822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.665797 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62"] Oct 31 13:36:17.849740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.837207 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-plugins-config-volume\") pod \"payload-processing-e2e-mt-e7896150-85d85d76df-vtc62\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:17.849740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.837344 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52znb\" (UniqueName: \"kubernetes.io/projected/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-kube-api-access-52znb\") pod \"payload-processing-e2e-mt-e7896150-85d85d76df-vtc62\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:17.849740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.837402 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwpl\" (UniqueName: \"kubernetes.io/projected/7af7844c-8037-47ed-8593-84a92852c243-kube-api-access-jjwpl\") pod \"payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:17.849740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.837566 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af7844c-8037-47ed-8593-84a92852c243-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:17.886656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.877745 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af7844c-8037-47ed-8593-84a92852c243-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:18.029471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.993597 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-plugins-config-volume\") pod \"payload-processing-e2e-mt-e7896150-85d85d76df-vtc62\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:18.029471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.993697 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52znb\" (UniqueName: \"kubernetes.io/projected/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-kube-api-access-52znb\") pod \"payload-processing-e2e-mt-e7896150-85d85d76df-vtc62\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:18.029471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:17.995304 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-plugins-config-volume\") pod \"payload-processing-e2e-mt-e7896150-85d85d76df-vtc62\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:18.039665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.038661 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwpl\" (UniqueName: \"kubernetes.io/projected/7af7844c-8037-47ed-8593-84a92852c243-kube-api-access-jjwpl\") pod \"payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:18.065476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.065440 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52znb\" (UniqueName: \"kubernetes.io/projected/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-kube-api-access-52znb\") pod \"payload-processing-e2e-mt-e7896150-85d85d76df-vtc62\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:18.100820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.100735 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c639106-3f06-4fe9-9883-f578170b9015" path="/var/lib/kubelet/pods/8c639106-3f06-4fe9-9883-f578170b9015/volumes" Oct 31 13:36:18.205648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.205599 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:18.285128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.283937 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:18.565222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.564420 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j"] Oct 31 13:36:18.565222 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:36:18.564769 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af7844c_8037_47ed_8593_84a92852c243.slice/crio-bd30a67fcccd1ce92c7fc2b0a7b956ec838479904d42375f14abec77a51544f5 WatchSource:0}: Error finding container bd30a67fcccd1ce92c7fc2b0a7b956ec838479904d42375f14abec77a51544f5: Status 404 returned error can't find the container with id bd30a67fcccd1ce92c7fc2b0a7b956ec838479904d42375f14abec77a51544f5 Oct 31 13:36:18.651399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.651371 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62"] Oct 31 13:36:18.653589 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:36:18.653548 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54bd3b0_5e0a_4b1c_95e6_53d65b44bb80.slice/crio-e23704f28ed923fdd1f3ed9d71975bdea87c527e297c212528125964f3e74066 WatchSource:0}: Error finding container e23704f28ed923fdd1f3ed9d71975bdea87c527e297c212528125964f3e74066: Status 404 returned error can't find the container with id e23704f28ed923fdd1f3ed9d71975bdea87c527e297c212528125964f3e74066 Oct 31 13:36:18.985356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.985314 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" event={"ID":"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80","Type":"ContainerStarted","Data":"4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e"} Oct 31 13:36:18.985549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.985388 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" event={"ID":"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80","Type":"ContainerStarted","Data":"e23704f28ed923fdd1f3ed9d71975bdea87c527e297c212528125964f3e74066"} Oct 31 13:36:18.985549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.985434 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:18.987478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.987447 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" event={"ID":"7af7844c-8037-47ed-8593-84a92852c243","Type":"ContainerStarted","Data":"8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee"} Oct 31 13:36:18.987601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.987508 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" event={"ID":"7af7844c-8037-47ed-8593-84a92852c243","Type":"ContainerStarted","Data":"bd30a67fcccd1ce92c7fc2b0a7b956ec838479904d42375f14abec77a51544f5"} Oct 31 13:36:18.987601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:18.987595 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:19.013939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:19.013881 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" podStartSLOduration=2.013866828 podStartE2EDuration="2.013866828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:19.012120096 +0000 UTC m=+3873.189838081" watchObservedRunningTime="2026-10-31 13:36:19.013866828 +0000 UTC m=+3873.191584814" Oct 31 13:36:19.042854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:19.042806 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" podStartSLOduration=2.04279371 podStartE2EDuration="2.04279371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:19.036807369 +0000 UTC m=+3873.214525354" watchObservedRunningTime="2026-10-31 13:36:19.04279371 +0000 UTC m=+3873.220511693" Oct 31 13:36:25.984985 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:36:25.984941 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:36:28.276932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.276882 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74"] Oct 31 13:36:28.283036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.282928 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.296440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.296411 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:36:28.304724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.296882 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:36:28.332812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.332781 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74"] Oct 31 13:36:28.439657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.432801 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6645\" (UniqueName: \"kubernetes.io/projected/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-kube-api-access-m6645\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.439657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.433015 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.439657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.433053 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.534978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.534881 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6645\" (UniqueName: \"kubernetes.io/projected/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-kube-api-access-m6645\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.535165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.535080 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.535165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.535121 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.535895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.535870 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.537547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.537522 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.547989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.547963 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6645\" (UniqueName: \"kubernetes.io/projected/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-kube-api-access-m6645\") pod \"maas-api-revoke-sub-c0f00da7-kpm74\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.627917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.627882 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:28.917088 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:36:28.917055 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef5ff65_f1c8_4cdd_8302_05fb77e02f29.slice/crio-c1836e1f7146b7bca412964b72ed1883cbcde7b20fb96676572b344850571636 WatchSource:0}: Error finding container c1836e1f7146b7bca412964b72ed1883cbcde7b20fb96676572b344850571636: Status 404 returned error can't find the container with id c1836e1f7146b7bca412964b72ed1883cbcde7b20fb96676572b344850571636 Oct 31 13:36:28.919697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:28.919664 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74"] Oct 31 13:36:29.073590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:29.073542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" event={"ID":"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29","Type":"ContainerStarted","Data":"c1836e1f7146b7bca412964b72ed1883cbcde7b20fb96676572b344850571636"} Oct 31 13:36:30.035657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:30.035218 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:36:30.040868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:30.040813 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:36:30.088067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:30.088029 5074 generic.go:356] "Generic (PLEG): container finished" podID="9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" containerID="91aa1739ab27008b4477f260cf42521370024f9849ca2f8419a250d1da67a512" exitCode=0 Oct 31 13:36:30.088283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:30.088260 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" event={"ID":"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29","Type":"ContainerDied","Data":"91aa1739ab27008b4477f260cf42521370024f9849ca2f8419a250d1da67a512"} Oct 31 13:36:31.302008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.301979 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:31.382214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.382122 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-kube-api-access-m6645\" (UniqueName: \"kubernetes.io/projected/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-kube-api-access-m6645\") pod \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " Oct 31 13:36:31.382512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.382341 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-xks-ca\") pod \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " Oct 31 13:36:31.382512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.382479 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-openshift-ca\") pod \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\" (UID: \"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29\") " Oct 31 13:36:31.383394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.383342 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-openshift-ca" pod "9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" (UID: "9ef5ff65-f1c8-4cdd-8302-05fb77e02f29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:31.384875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.384847 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-kube-api-access-m6645" pod "9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" (UID: "9ef5ff65-f1c8-4cdd-8302-05fb77e02f29"). InnerVolumeSpecName "kube-api-access-m6645". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:31.385015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.384991 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-xks-ca" pod "9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" (UID: "9ef5ff65-f1c8-4cdd-8302-05fb77e02f29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:31.484349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.484315 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6645\" (UniqueName: \"kubernetes.io/projected/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-kube-api-access-m6645\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:31.484349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.484347 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:31.484551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:31.484357 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:32.106282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:32.106247 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" Oct 31 13:36:32.106470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:32.106280 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74" event={"ID":"9ef5ff65-f1c8-4cdd-8302-05fb77e02f29","Type":"ContainerDied","Data":"c1836e1f7146b7bca412964b72ed1883cbcde7b20fb96676572b344850571636"} Oct 31 13:36:32.106470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:32.106339 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1836e1f7146b7bca412964b72ed1883cbcde7b20fb96676572b344850571636" Oct 31 13:36:33.586768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.586727 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl"] Oct 31 13:36:33.587936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.587884 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" containerName="revoke-keys" Oct 31 13:36:33.587936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.587911 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" containerName="revoke-keys" Oct 31 13:36:33.588166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.588112 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" containerName="revoke-keys" Oct 31 13:36:33.592281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.592257 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.597741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.597622 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:36:33.598690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.598663 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:36:33.606997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.603194 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl"] Oct 31 13:36:33.630650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.630606 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.630870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.630843 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.631051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.631033 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hkc\" (UniqueName: \"kubernetes.io/projected/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-kube-api-access-t5hkc\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.732698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.732658 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hkc\" (UniqueName: \"kubernetes.io/projected/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-kube-api-access-t5hkc\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.732910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.732782 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.732981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.732943 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.733482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.733451 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.735453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.735432 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.745281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.745257 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hkc\" (UniqueName: \"kubernetes.io/projected/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-kube-api-access-t5hkc\") pod \"maas-api-revoke-sub-4732e9d5-zbckl\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:33.922597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:33.922506 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:34.288812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:34.288771 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl"] Oct 31 13:36:34.289410 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:36:34.289364 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6329f68a_deb4_4bb7_bd8d_d7ddc805a479.slice/crio-48e125df4674af086e2a12dce30fc5a19804ae032f2ad9d9720fb1943ed8490e WatchSource:0}: Error finding container 48e125df4674af086e2a12dce30fc5a19804ae032f2ad9d9720fb1943ed8490e: Status 404 returned error can't find the container with id 48e125df4674af086e2a12dce30fc5a19804ae032f2ad9d9720fb1943ed8490e Oct 31 13:36:35.136512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:35.136412 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" event={"ID":"6329f68a-deb4-4bb7-bd8d-d7ddc805a479","Type":"ContainerStarted","Data":"3f1ad1295de7d738c4480c0942d664b0189fa45b902e1ea2793ebb379f10be04"} Oct 31 13:36:35.136512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:35.136479 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" event={"ID":"6329f68a-deb4-4bb7-bd8d-d7ddc805a479","Type":"ContainerStarted","Data":"48e125df4674af086e2a12dce30fc5a19804ae032f2ad9d9720fb1943ed8490e"} Oct 31 13:36:35.158305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:35.158255 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" podStartSLOduration=2.158241552 podStartE2EDuration="2.158241552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:35.156869371 +0000 UTC m=+3889.334587354" watchObservedRunningTime="2026-10-31 13:36:35.158241552 +0000 UTC m=+3889.335959536" Oct 31 13:36:36.161356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:36.145705 5074 generic.go:356] "Generic (PLEG): container finished" podID="6329f68a-deb4-4bb7-bd8d-d7ddc805a479" containerID="3f1ad1295de7d738c4480c0942d664b0189fa45b902e1ea2793ebb379f10be04" exitCode=0 Oct 31 13:36:36.161356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:36.145805 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" event={"ID":"6329f68a-deb4-4bb7-bd8d-d7ddc805a479","Type":"ContainerDied","Data":"3f1ad1295de7d738c4480c0942d664b0189fa45b902e1ea2793ebb379f10be04"} Oct 31 13:36:37.344437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.344408 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:37.432434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.432347 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-openshift-ca\") pod \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " Oct 31 13:36:37.432737 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.432713 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-openshift-ca" pod "6329f68a-deb4-4bb7-bd8d-d7ddc805a479" (UID: "6329f68a-deb4-4bb7-bd8d-d7ddc805a479"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:37.432806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.432731 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-xks-ca\") pod \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " Oct 31 13:36:37.432806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.432772 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-kube-api-access-t5hkc\" (UniqueName: \"kubernetes.io/projected/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-kube-api-access-t5hkc\") pod \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\" (UID: \"6329f68a-deb4-4bb7-bd8d-d7ddc805a479\") " Oct 31 13:36:37.433616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.433588 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:37.434857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.434830 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-xks-ca" pod "6329f68a-deb4-4bb7-bd8d-d7ddc805a479" (UID: "6329f68a-deb4-4bb7-bd8d-d7ddc805a479"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:37.435011 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.434986 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-kube-api-access-t5hkc" pod "6329f68a-deb4-4bb7-bd8d-d7ddc805a479" (UID: "6329f68a-deb4-4bb7-bd8d-d7ddc805a479"). InnerVolumeSpecName "kube-api-access-t5hkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:37.535077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.535038 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:37.535077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:37.535072 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hkc\" (UniqueName: \"kubernetes.io/projected/6329f68a-deb4-4bb7-bd8d-d7ddc805a479-kube-api-access-t5hkc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:38.163092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:38.163058 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" Oct 31 13:36:38.163092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:38.163085 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl" event={"ID":"6329f68a-deb4-4bb7-bd8d-d7ddc805a479","Type":"ContainerDied","Data":"48e125df4674af086e2a12dce30fc5a19804ae032f2ad9d9720fb1943ed8490e"} Oct 31 13:36:38.163359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:38.163133 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e125df4674af086e2a12dce30fc5a19804ae032f2ad9d9720fb1943ed8490e" Oct 31 13:36:39.894655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:39.893987 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6"] Oct 31 13:36:39.906681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:39.895277 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6329f68a-deb4-4bb7-bd8d-d7ddc805a479" containerName="revoke-keys" Oct 31 13:36:39.906681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:39.895303 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6329f68a-deb4-4bb7-bd8d-d7ddc805a479" containerName="revoke-keys" Oct 31 13:36:39.906681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:39.895524 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6329f68a-deb4-4bb7-bd8d-d7ddc805a479" containerName="revoke-keys" Oct 31 13:36:39.906681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:39.900459 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:39.912476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:39.908919 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e7896150-serving-cert\"" Oct 31 13:36:39.912476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:39.910759 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6"] Oct 31 13:36:40.098661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.091027 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-maas-api-tls\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.098661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.091219 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqkn\" (UniqueName: \"kubernetes.io/projected/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-kube-api-access-6bqkn\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.098661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.091282 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-metrics-certs\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.098661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.091403 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-openshift-service-ca\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.198137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.198041 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqkn\" (UniqueName: \"kubernetes.io/projected/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-kube-api-access-6bqkn\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.198433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.198410 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-metrics-certs\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.198696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.198677 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-openshift-service-ca\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.205947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.205921 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-maas-api-tls\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.206288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.200377 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-openshift-service-ca\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.222650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.220524 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-maas-api-tls\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.222650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.222070 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-metrics-certs\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.223365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.223333 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqkn\" (UniqueName: \"kubernetes.io/projected/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-kube-api-access-6bqkn\") pod \"maas-api-e2e-mt-e7896150-86457b6d97-wpdm6\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:40.535197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:40.535158 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:41.043075 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:36:41.043047 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d7431e_9744_41b5_9c3e_25d4bdd552fa.slice/crio-0362609d49e1489145f98721ba4c7edb4ed6c69bae3c9c89710f48a176af1752 WatchSource:0}: Error finding container 0362609d49e1489145f98721ba4c7edb4ed6c69bae3c9c89710f48a176af1752: Status 404 returned error can't find the container with id 0362609d49e1489145f98721ba4c7edb4ed6c69bae3c9c89710f48a176af1752 Oct 31 13:36:41.045796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:41.045766 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6"] Oct 31 13:36:41.046181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:41.046152 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:36:41.230783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:41.230743 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" event={"ID":"f4d7431e-9744-41b5-9c3e-25d4bdd552fa","Type":"ContainerStarted","Data":"0362609d49e1489145f98721ba4c7edb4ed6c69bae3c9c89710f48a176af1752"} Oct 31 13:36:42.240016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:42.239979 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" event={"ID":"f4d7431e-9744-41b5-9c3e-25d4bdd552fa","Type":"ContainerStarted","Data":"54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418"} Oct 31 13:36:42.240435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:42.240039 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:42.264413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:42.264359 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" podStartSLOduration=2.695553212 podStartE2EDuration="3.264344548s" totalImagesPullingTime="568.791336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:42.260109104 +0000 UTC m=+3896.437827091" watchObservedRunningTime="2026-10-31 13:36:42.264344548 +0000 UTC m=+3896.442062531" Oct 31 13:36:44.671318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.671279 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n"] Oct 31 13:36:44.677438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.677408 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.684493 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.684464 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:36:44.685056 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.684512 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:36:44.686186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.686156 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n"] Oct 31 13:36:44.736878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.736839 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9npd\" (UniqueName: \"kubernetes.io/projected/25263f3e-949f-459c-81b1-494cd89a19c1-kube-api-access-r9npd\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.737092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.737051 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.737281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.737247 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.838744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.838705 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.838934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.838849 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.838934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.838893 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9npd\" (UniqueName: \"kubernetes.io/projected/25263f3e-949f-459c-81b1-494cd89a19c1-kube-api-access-r9npd\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.839526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.839501 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.841311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.841284 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.851327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.851301 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9npd\" (UniqueName: \"kubernetes.io/projected/25263f3e-949f-459c-81b1-494cd89a19c1-kube-api-access-r9npd\") pod \"maas-api-revoke-sub-cc102b33-4x59n\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:44.998645 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:44.998600 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:45.398829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:45.398697 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n"] Oct 31 13:36:45.400480 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:36:45.400451 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25263f3e_949f_459c_81b1_494cd89a19c1.slice/crio-976bc16ab0b7e19ee4058fa9497831b4d64080ec35cf21ad1d1dbb5afe2f2f51 WatchSource:0}: Error finding container 976bc16ab0b7e19ee4058fa9497831b4d64080ec35cf21ad1d1dbb5afe2f2f51: Status 404 returned error can't find the container with id 976bc16ab0b7e19ee4058fa9497831b4d64080ec35cf21ad1d1dbb5afe2f2f51 Oct 31 13:36:46.283130 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:46.283095 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" event={"ID":"25263f3e-949f-459c-81b1-494cd89a19c1","Type":"ContainerStarted","Data":"0afe02d4cb78ad9ffd527b63c53336fb925bc676ba68eb502577081caee96dcc"} Oct 31 13:36:46.283655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:46.283149 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" event={"ID":"25263f3e-949f-459c-81b1-494cd89a19c1","Type":"ContainerStarted","Data":"976bc16ab0b7e19ee4058fa9497831b4d64080ec35cf21ad1d1dbb5afe2f2f51"} Oct 31 13:36:46.354917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:46.354848 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" podStartSLOduration=2.3548337679999998 podStartE2EDuration="2.354833768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:46.351272294 +0000 UTC m=+3900.528990276" watchObservedRunningTime="2026-10-31 13:36:46.354833768 +0000 UTC m=+3900.532551752" Oct 31 13:36:47.134019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:47.133979 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:36:47.156213 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:47.156186 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:36:47.295300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:47.295263 5074 generic.go:356] "Generic (PLEG): container finished" podID="25263f3e-949f-459c-81b1-494cd89a19c1" containerID="0afe02d4cb78ad9ffd527b63c53336fb925bc676ba68eb502577081caee96dcc" exitCode=0 Oct 31 13:36:47.295788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:47.295353 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" event={"ID":"25263f3e-949f-459c-81b1-494cd89a19c1","Type":"ContainerDied","Data":"0afe02d4cb78ad9ffd527b63c53336fb925bc676ba68eb502577081caee96dcc"} Oct 31 13:36:48.253950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.253922 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:36:48.562879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.562850 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:48.726208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.726170 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-xks-ca\") pod \"25263f3e-949f-459c-81b1-494cd89a19c1\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " Oct 31 13:36:48.726413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.726397 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-openshift-ca\") pod \"25263f3e-949f-459c-81b1-494cd89a19c1\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " Oct 31 13:36:48.726546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.726524 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25263f3e-949f-459c-81b1-494cd89a19c1-kube-api-access-r9npd\" (UniqueName: \"kubernetes.io/projected/25263f3e-949f-459c-81b1-494cd89a19c1-kube-api-access-r9npd\") pod \"25263f3e-949f-459c-81b1-494cd89a19c1\" (UID: \"25263f3e-949f-459c-81b1-494cd89a19c1\") " Oct 31 13:36:48.726868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.726836 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-openshift-ca" pod "25263f3e-949f-459c-81b1-494cd89a19c1" (UID: "25263f3e-949f-459c-81b1-494cd89a19c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:48.727089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.727073 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:48.728403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.728374 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-xks-ca" pod "25263f3e-949f-459c-81b1-494cd89a19c1" (UID: "25263f3e-949f-459c-81b1-494cd89a19c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:48.728784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.728755 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25263f3e-949f-459c-81b1-494cd89a19c1-kube-api-access-r9npd" pod "25263f3e-949f-459c-81b1-494cd89a19c1" (UID: "25263f3e-949f-459c-81b1-494cd89a19c1"). InnerVolumeSpecName "kube-api-access-r9npd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:48.828387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.828306 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9npd\" (UniqueName: \"kubernetes.io/projected/25263f3e-949f-459c-81b1-494cd89a19c1-kube-api-access-r9npd\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:48.828387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:48.828341 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25263f3e-949f-459c-81b1-494cd89a19c1-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:49.319371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:49.319338 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" Oct 31 13:36:49.319371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:49.319358 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n" event={"ID":"25263f3e-949f-459c-81b1-494cd89a19c1","Type":"ContainerDied","Data":"976bc16ab0b7e19ee4058fa9497831b4d64080ec35cf21ad1d1dbb5afe2f2f51"} Oct 31 13:36:49.319687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:49.319419 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="976bc16ab0b7e19ee4058fa9497831b4d64080ec35cf21ad1d1dbb5afe2f2f51" Oct 31 13:36:49.981922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:49.981886 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:36:55.692702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:36:55.692660 5074 scope.go:122] "RemoveContainer" containerID="ca6e7d007d23aacc315a573df2c53ff94d776477b9a16649fc6a669b5e4d28b8" Oct 31 13:36:55.991627 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:36:55.991585 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:37:06.676771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:06.676569 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp"] Oct 31 13:37:06.704894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:06.676846 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" podUID="b9544b2c-8723-464e-88e1-bb656e81b058" containerName="maas-api" containerID="cri-o://0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e" gracePeriod=30 Oct 31 13:37:07.401236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.401200 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:37:07.522342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.522302 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-maas-api-tls\") pod \"b9544b2c-8723-464e-88e1-bb656e81b058\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " Oct 31 13:37:07.522514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.522444 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9544b2c-8723-464e-88e1-bb656e81b058-kube-api-access-vl6nb\" (UniqueName: \"kubernetes.io/projected/b9544b2c-8723-464e-88e1-bb656e81b058-kube-api-access-vl6nb\") pod \"b9544b2c-8723-464e-88e1-bb656e81b058\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " Oct 31 13:37:07.522514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.522490 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-metrics-certs\") pod \"b9544b2c-8723-464e-88e1-bb656e81b058\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " Oct 31 13:37:07.522684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.522610 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9544b2c-8723-464e-88e1-bb656e81b058-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9544b2c-8723-464e-88e1-bb656e81b058-openshift-service-ca\") pod \"b9544b2c-8723-464e-88e1-bb656e81b058\" (UID: \"b9544b2c-8723-464e-88e1-bb656e81b058\") " Oct 31 13:37:07.523044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.523011 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9544b2c-8723-464e-88e1-bb656e81b058-openshift-service-ca" pod "b9544b2c-8723-464e-88e1-bb656e81b058" (UID: "b9544b2c-8723-464e-88e1-bb656e81b058"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:07.523461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.523431 5074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9544b2c-8723-464e-88e1-bb656e81b058-openshift-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.524674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.524619 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-maas-api-tls" pod "b9544b2c-8723-464e-88e1-bb656e81b058" (UID: "b9544b2c-8723-464e-88e1-bb656e81b058"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:07.524858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.524829 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9544b2c-8723-464e-88e1-bb656e81b058-kube-api-access-vl6nb" pod "b9544b2c-8723-464e-88e1-bb656e81b058" (UID: "b9544b2c-8723-464e-88e1-bb656e81b058"). InnerVolumeSpecName "kube-api-access-vl6nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:07.524957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.524894 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-metrics-certs" pod "b9544b2c-8723-464e-88e1-bb656e81b058" (UID: "b9544b2c-8723-464e-88e1-bb656e81b058"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:07.597765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.597671 5074 generic.go:356] "Generic (PLEG): container finished" podID="b9544b2c-8723-464e-88e1-bb656e81b058" containerID="0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e" exitCode=0 Oct 31 13:37:07.597765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.597724 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" Oct 31 13:37:07.597765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.597746 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" event={"ID":"b9544b2c-8723-464e-88e1-bb656e81b058","Type":"ContainerDied","Data":"0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e"} Oct 31 13:37:07.598025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.597812 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp" event={"ID":"b9544b2c-8723-464e-88e1-bb656e81b058","Type":"ContainerDied","Data":"6ef10e2523f08e599fb60f322d267a659cebc120a99e1194522db6056b2edd54"} Oct 31 13:37:07.598025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.597841 5074 scope.go:122] "RemoveContainer" containerID="0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e" Oct 31 13:37:07.610401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.610369 5074 scope.go:122] "RemoveContainer" containerID="0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e" Oct 31 13:37:07.610714 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:07.610688 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e\": container with ID starting with 0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e not found: ID does not exist" containerID="0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e" Oct 31 13:37:07.610812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.610727 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e"} err="failed to get container status \"0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e\": rpc error: code = NotFound desc = could not find container \"0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e\": container with ID starting with 0b8d72c6f086655171656a72aed57815a9c59062caa1065af5df64afa1ed072e not found: ID does not exist" Oct 31 13:37:07.625100 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.625068 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl6nb\" (UniqueName: \"kubernetes.io/projected/b9544b2c-8723-464e-88e1-bb656e81b058-kube-api-access-vl6nb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.625227 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.625105 5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-metrics-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.625227 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.625121 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9544b2c-8723-464e-88e1-bb656e81b058-maas-api-tls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.631533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.631477 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp"] Oct 31 13:37:07.636698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:07.636674 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-691f1e-69c98bbd88-xw5kp"] Oct 31 13:37:08.029267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:08.029233 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9544b2c-8723-464e-88e1-bb656e81b058" path="/var/lib/kubelet/pods/b9544b2c-8723-464e-88e1-bb656e81b058/volumes" Oct 31 13:37:11.522109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:11.522063 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c"] Oct 31 13:37:11.522587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:11.522307 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" containerID="cri-o://745177e4357c15e36b7ce09b6a06b6d472b226a115c88cee66b799e96d436d60" gracePeriod=30 Oct 31 13:37:13.941204 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:13.941171 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k"] Oct 31 13:37:13.941658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:13.941366 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" containerID="cri-o://df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6" gracePeriod=30 Oct 31 13:37:14.291372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:14.291335 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx"] Oct 31 13:37:14.291584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:14.291550 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" containerID="cri-o://eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591" gracePeriod=30 Oct 31 13:37:14.885461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:14.885420 5074 patch_prober.go:29] interesting pod/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:37:14.885661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:14.885483 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:37:15.601698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:15.600311 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v"] Oct 31 13:37:15.601698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:15.600595 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" containerID="cri-o://aaaf8c08c1d00e8c8e89baa3b6cb1f1f870cd6bd9c5ca866cfbf7bc468ab3191" gracePeriod=30 Oct 31 13:37:16.706977 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.706738 5074 generic.go:356] "Generic (PLEG): container finished" podID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerID="745177e4357c15e36b7ce09b6a06b6d472b226a115c88cee66b799e96d436d60" exitCode=0 Oct 31 13:37:16.707328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.707044 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" event={"ID":"80e07e60-95d3-4a6c-8838-4789a48ff2d6","Type":"ContainerDied","Data":"745177e4357c15e36b7ce09b6a06b6d472b226a115c88cee66b799e96d436d60"} Oct 31 13:37:16.833311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.833281 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 13:37:16.884803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.884762 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-data\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.884997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.884820 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-credential-socket\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.884997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.884904 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-token\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-token\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.884997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.884939 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-podinfo\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.885196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885104 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-kube-api-access-gvgnk\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-kube-api-access-gvgnk\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.885196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885166 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-credential-socket" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.885308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885230 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istiod-ca-cert\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.885308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885284 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-socket\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.885425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885338 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-envoy\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.885425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885385 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-certs\") pod \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\" (UID: \"80e07e60-95d3-4a6c-8838-4789a48ff2d6\") " Oct 31 13:37:16.885776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885748 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-data" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.886017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.885993 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-certs" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.886410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.886387 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.886601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.886586 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.886800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.886784 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.886892 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.886505 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istiod-ca-cert" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:16.886968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.886769 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-socket" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.895091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.895063 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-envoy" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:16.909256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.909220 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-token" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:16.911102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.911078 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-kube-api-access-gvgnk" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "kube-api-access-gvgnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:16.911276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.911253 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-podinfo" pod "80e07e60-95d3-4a6c-8838-4789a48ff2d6" (UID: "80e07e60-95d3-4a6c-8838-4789a48ff2d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:16.988394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.988359 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.988394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.988396 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.988593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.988411 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvgnk\" (UniqueName: \"kubernetes.io/projected/80e07e60-95d3-4a6c-8838-4789a48ff2d6-kube-api-access-gvgnk\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.988593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.988427 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.988593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.988442 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:16.988593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:16.988456 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e07e60-95d3-4a6c-8838-4789a48ff2d6-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:17.333931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.333893 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx"] Oct 31 13:37:17.334667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334651 5074 cpu_manager.go:414] "containerMap: removing container" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" Oct 31 13:37:17.334715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334669 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" Oct 31 13:37:17.334715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334703 5074 cpu_manager.go:414] "containerMap: removing container" podUID="25263f3e-949f-459c-81b1-494cd89a19c1" containerName="revoke-keys" Oct 31 13:37:17.334715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334709 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25263f3e-949f-459c-81b1-494cd89a19c1" containerName="revoke-keys" Oct 31 13:37:17.334867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334729 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b9544b2c-8723-464e-88e1-bb656e81b058" containerName="maas-api" Oct 31 13:37:17.334867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334736 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9544b2c-8723-464e-88e1-bb656e81b058" containerName="maas-api" Oct 31 13:37:17.334943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334886 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" containerName="istio-proxy" Oct 31 13:37:17.334943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334903 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="25263f3e-949f-459c-81b1-494cd89a19c1" containerName="revoke-keys" Oct 31 13:37:17.334943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.334913 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9544b2c-8723-464e-88e1-bb656e81b058" containerName="maas-api" Oct 31 13:37:17.339928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.339909 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.344568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.344541 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:37:17.344797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.344671 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:37:17.349330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.349301 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx"] Oct 31 13:37:17.397064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.397029 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.397269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.397128 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.397269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.397260 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlw42\" (UniqueName: \"kubernetes.io/projected/6c0bc38b-efce-493f-aa8b-4c4c28181384-kube-api-access-wlw42\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.498858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.498815 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.499065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.498913 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.499065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.499030 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlw42\" (UniqueName: \"kubernetes.io/projected/6c0bc38b-efce-493f-aa8b-4c4c28181384-kube-api-access-wlw42\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.499616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.499592 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.501323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.501301 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.511594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.511568 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlw42\" (UniqueName: \"kubernetes.io/projected/6c0bc38b-efce-493f-aa8b-4c4c28181384-kube-api-access-wlw42\") pod \"maas-api-revoke-sub-275549df-5p8wx\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.663548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.663462 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:17.720236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.718074 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" event={"ID":"80e07e60-95d3-4a6c-8838-4789a48ff2d6","Type":"ContainerDied","Data":"8ad45b62f5a1d569cce8ac2a1e6a9d59b9892ca6085fb67a81c7d609e5a1ba46"} Oct 31 13:37:17.720236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.718147 5074 scope.go:122] "RemoveContainer" containerID="745177e4357c15e36b7ce09b6a06b6d472b226a115c88cee66b799e96d436d60" Oct 31 13:37:17.720236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.718308 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c" Oct 31 13:37:17.759254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.759222 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c"] Oct 31 13:37:17.764669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.763885 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ae9e7-openshift-default-7b486489d5-mjx7c"] Oct 31 13:37:17.942026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.941937 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e07e60-95d3-4a6c-8838-4789a48ff2d6" path="/var/lib/kubelet/pods/80e07e60-95d3-4a6c-8838-4789a48ff2d6/volumes" Oct 31 13:37:17.961936 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:37:17.961904 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0bc38b_efce_493f_aa8b_4c4c28181384.slice/crio-d4c40972ff686c934ec3df913e5334849620d3991d9f09702484cbf3a153160f WatchSource:0}: Error finding container d4c40972ff686c934ec3df913e5334849620d3991d9f09702484cbf3a153160f: Status 404 returned error can't find the container with id d4c40972ff686c934ec3df913e5334849620d3991d9f09702484cbf3a153160f Oct 31 13:37:17.962105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:17.962083 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx"] Oct 31 13:37:18.007397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.006414 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6"] Oct 31 13:37:18.309666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.301105 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:37:18.309666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.301170 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:37:18.439658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.422724 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc"] Oct 31 13:37:18.470683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.461938 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.473232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.473197 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-50b4aa-openshift-default-dockercfg-5fzkn\"" Oct 31 13:37:18.474267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.474241 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc"] Oct 31 13:37:18.536045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.536008 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-data\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.536319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.536277 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-istio-token\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.536753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.536717 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5kf7\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-kube-api-access-l5kf7\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.536927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.536910 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601212db-06e4-49b6-b027-5d37562ca8c2-istio-podinfo\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.537155 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.537138 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-envoy\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.537453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.537437 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-credential-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.537591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.537576 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601212db-06e4-49b6-b027-5d37562ca8c2-istiod-ca-cert\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.537719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.537705 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.541953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.541925 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-certs\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649032 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-certs\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649110 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-data\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649144 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-istio-token\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649279 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5kf7\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-kube-api-access-l5kf7\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649310 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601212db-06e4-49b6-b027-5d37562ca8c2-istio-podinfo\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649345 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-envoy\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649529 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-credential-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649594 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601212db-06e4-49b6-b027-5d37562ca8c2-istiod-ca-cert\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.649709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.649623 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.650859 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.650153 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.651046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.651022 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-certs\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.651394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.651314 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-data\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.660437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.659587 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-credential-socket\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.669219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.668268 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-envoy\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.669521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.669500 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601212db-06e4-49b6-b027-5d37562ca8c2-istiod-ca-cert\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.685190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.685146 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601212db-06e4-49b6-b027-5d37562ca8c2-istio-podinfo\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.689189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.688833 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5kf7\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-kube-api-access-l5kf7\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.695483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.695463 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-istio-token\") pod \"e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:18.747767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.747720 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" event={"ID":"6c0bc38b-efce-493f-aa8b-4c4c28181384","Type":"ContainerStarted","Data":"68871357b20a5e22a78116af5bdab354677b1b2ba842e4ec11428ec22dc33d02"} Oct 31 13:37:18.748278 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.747789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" event={"ID":"6c0bc38b-efce-493f-aa8b-4c4c28181384","Type":"ContainerStarted","Data":"d4c40972ff686c934ec3df913e5334849620d3991d9f09702484cbf3a153160f"} Oct 31 13:37:18.793433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.793381 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" podStartSLOduration=1.793365877 podStartE2EDuration="1.793365877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:18.78899164 +0000 UTC m=+3932.966709627" watchObservedRunningTime="2026-10-31 13:37:18.793365877 +0000 UTC m=+3932.971083858" Oct 31 13:37:18.857321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:18.857287 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:19.216681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.214622 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc"] Oct 31 13:37:19.245229 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:37:19.244524 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601212db_06e4_49b6_b027_5d37562ca8c2.slice/crio-1761d0da9fda276abb5aace25f540b94d09cac87bb412daed358bc539e5103f8 WatchSource:0}: Error finding container 1761d0da9fda276abb5aace25f540b94d09cac87bb412daed358bc539e5103f8: Status 404 returned error can't find the container with id 1761d0da9fda276abb5aace25f540b94d09cac87bb412daed358bc539e5103f8 Oct 31 13:37:19.248313 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.248275 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:19.248453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.248348 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:19.248453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.248377 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:19.249334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.249313 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:37:19.382933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.382888 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istiod-ca-cert\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.382960 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-podinfo\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.382993 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-envoy\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.383049 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-kube-api-access-wp84k\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-kube-api-access-wp84k\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.383184 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-credential-socket\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.383273 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-data\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.383320 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-certs\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.383413 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-socket\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.383504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.383441 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-token\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-token\") pod \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\" (UID: \"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e\") " Oct 31 13:37:19.386089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.385873 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-certs" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.386818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.386338 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-credential-socket" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.387218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.387008 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-socket" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.387507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.387465 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istiod-ca-cert" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:19.387880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.387859 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-data" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.388894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.388861 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-podinfo" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:19.389272 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.389239 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-envoy" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.389391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.389371 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-kube-api-access-wp84k" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "kube-api-access-wp84k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:19.390837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.390810 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-token" pod "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" (UID: "391f2b95-3fd5-41cb-867d-74f0bfa8cc2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:19.508655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508491 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.508655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508539 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.508655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508556 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.508655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508570 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.508655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508586 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.508655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508599 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.508655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508613 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.509178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508626 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.509178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.508688 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp84k\" (UniqueName: \"kubernetes.io/projected/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e-kube-api-access-wp84k\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.716258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.716230 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:37:19.717058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.717032 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-socket\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.717171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.717134 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-kube-api-access-4kzvn\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-kube-api-access-4kzvn\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.717245 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.717185 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18c2078-ee42-451c-9685-1e6f118ac9ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18c2078-ee42-451c-9685-1e6f118ac9ec-istiod-ca-cert\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.717333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.717301 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-socket" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.717730 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.717708 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18c2078-ee42-451c-9685-1e6f118ac9ec-istiod-ca-cert" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:19.717969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.717941 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.717969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.717961 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18c2078-ee42-451c-9685-1e6f118ac9ec-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.719602 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.719583 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-kube-api-access-4kzvn" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "kube-api-access-4kzvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:19.759771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.759688 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" event={"ID":"601212db-06e4-49b6-b027-5d37562ca8c2","Type":"ContainerStarted","Data":"4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc"} Oct 31 13:37:19.759771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.759750 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" event={"ID":"601212db-06e4-49b6-b027-5d37562ca8c2","Type":"ContainerStarted","Data":"1761d0da9fda276abb5aace25f540b94d09cac87bb412daed358bc539e5103f8"} Oct 31 13:37:19.764655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.764122 5074 generic.go:356] "Generic (PLEG): container finished" podID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerID="df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6" exitCode=0 Oct 31 13:37:19.764655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.764279 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" event={"ID":"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e","Type":"ContainerDied","Data":"df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6"} Oct 31 13:37:19.764655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.764326 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" event={"ID":"391f2b95-3fd5-41cb-867d-74f0bfa8cc2e","Type":"ContainerDied","Data":"758e2488a08939ac3593e5b8f3e9466b2d59f9fc7dc0dc6a684483ca3b01cc46"} Oct 31 13:37:19.764655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.764355 5074 scope.go:122] "RemoveContainer" containerID="df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6" Oct 31 13:37:19.764655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.764529 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k" Oct 31 13:37:19.769173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.769148 5074 generic.go:356] "Generic (PLEG): container finished" podID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerID="eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591" exitCode=0 Oct 31 13:37:19.769302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.769243 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" event={"ID":"c18c2078-ee42-451c-9685-1e6f118ac9ec","Type":"ContainerDied","Data":"eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591"} Oct 31 13:37:19.769379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.769311 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" event={"ID":"c18c2078-ee42-451c-9685-1e6f118ac9ec","Type":"ContainerDied","Data":"91e7131edae04fd17c279e7a07e39e002078dc48c0110f2e9b159633a769872c"} Oct 31 13:37:19.769446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.769401 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx" Oct 31 13:37:19.771545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.771520 5074 generic.go:356] "Generic (PLEG): container finished" podID="6c0bc38b-efce-493f-aa8b-4c4c28181384" containerID="68871357b20a5e22a78116af5bdab354677b1b2ba842e4ec11428ec22dc33d02" exitCode=0 Oct 31 13:37:19.771671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.771573 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" event={"ID":"6c0bc38b-efce-493f-aa8b-4c4c28181384","Type":"ContainerDied","Data":"68871357b20a5e22a78116af5bdab354677b1b2ba842e4ec11428ec22dc33d02"} Oct 31 13:37:19.781332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.781308 5074 scope.go:122] "RemoveContainer" containerID="df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6" Oct 31 13:37:19.781679 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:19.781622 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6\": container with ID starting with df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6 not found: ID does not exist" containerID="df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6" Oct 31 13:37:19.781756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.781689 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6"} err="failed to get container status \"df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6\": rpc error: code = NotFound desc = could not find container \"df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6\": container with ID starting with df91781aea56c7d26473ee69b5e40b26cdffaa18745e4d14f50b4a3a84245ba6 not found: ID does not exist" Oct 31 13:37:19.781756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.781708 5074 scope.go:122] "RemoveContainer" containerID="eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591" Oct 31 13:37:19.802826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.802794 5074 scope.go:122] "RemoveContainer" containerID="eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591" Oct 31 13:37:19.803413 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:19.803383 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591\": container with ID starting with eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591 not found: ID does not exist" containerID="eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591" Oct 31 13:37:19.803511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.803425 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591"} err="failed to get container status \"eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591\": rpc error: code = NotFound desc = could not find container \"eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591\": container with ID starting with eb50995ba0f1e4e3cbd65c0cd854f9625164c10906e24c901a3c48e41991b591 not found: ID does not exist" Oct 31 13:37:19.818866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.818838 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-podinfo\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.819250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.819229 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-certs\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.819419 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.819401 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-envoy\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.819556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.819530 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-certs" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.819680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.819534 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-token\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-token\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.819767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.819747 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-data\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.819835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.819791 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-credential-socket\") pod \"c18c2078-ee42-451c-9685-1e6f118ac9ec\" (UID: \"c18c2078-ee42-451c-9685-1e6f118ac9ec\") " Oct 31 13:37:19.821285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.820556 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kzvn\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-kube-api-access-4kzvn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.821285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.820585 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.821285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.821208 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-credential-socket" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.821285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.821263 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-data" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.822351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.822320 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-podinfo" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:19.823907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.823886 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-token" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:19.824557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.824504 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-envoy" pod "c18c2078-ee42-451c-9685-1e6f118ac9ec" (UID: "c18c2078-ee42-451c-9685-1e6f118ac9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:19.857959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.857926 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:19.894676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.892903 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k"] Oct 31 13:37:19.898440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.898411 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-7cdec6-openshift-default-754d58877f-rrp9k"] Oct 31 13:37:19.926667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.926607 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.927615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.927588 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.927789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.927775 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.927880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.927868 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.927970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.927958 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18c2078-ee42-451c-9685-1e6f118ac9ec-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:19.933012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.931727 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podStartSLOduration=1.931711266 podStartE2EDuration="1.931711266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:19.925042715 +0000 UTC m=+3934.102760699" watchObservedRunningTime="2026-10-31 13:37:19.931711266 +0000 UTC m=+3934.109429251" Oct 31 13:37:19.946590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:19.946552 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" path="/var/lib/kubelet/pods/391f2b95-3fd5-41cb-867d-74f0bfa8cc2e/volumes" Oct 31 13:37:20.101790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.101700 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx"] Oct 31 13:37:20.101972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.101878 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5dc570-openshift-default-7fb985cdd5-m64fx"] Oct 31 13:37:20.339439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.339402 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:37:20.339624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.339458 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:37:20.822202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.812796 5074 generic.go:356] "Generic (PLEG): container finished" podID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerID="aaaf8c08c1d00e8c8e89baa3b6cb1f1f870cd6bd9c5ca866cfbf7bc468ab3191" exitCode=0 Oct 31 13:37:20.822202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.812918 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" event={"ID":"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf","Type":"ContainerDied","Data":"aaaf8c08c1d00e8c8e89baa3b6cb1f1f870cd6bd9c5ca866cfbf7bc468ab3191"} Oct 31 13:37:20.860554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.860192 5074 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 13:37:20.860554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.860254 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 13:37:20.964917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:20.964861 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:37:21.046107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046068 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istiod-ca-cert\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.046310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046152 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-credential-socket\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.046310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046212 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-kube-api-access-p48bs\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-kube-api-access-p48bs\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.046310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046285 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-podinfo\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.046476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046437 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-envoy\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.046543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046485 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-credential-socket" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.046543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046509 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istiod-ca-cert" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:21.046543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046499 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-certs\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.046722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046602 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-token\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-token\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.046779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046753 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-certs" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.046830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046790 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-socket\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.047008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.046985 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-data\") pod \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\" (UID: \"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf\") " Oct 31 13:37:21.048893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.048837 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.048893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.048871 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.049094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.049070 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-data" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.051597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.051561 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-socket" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.053499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.053453 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-envoy" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:21.053839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.053797 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-kube-api-access-p48bs" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "kube-api-access-p48bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:21.055196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.055162 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-token" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:21.060865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.060813 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-podinfo" pod "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" (UID: "330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:21.146318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.146288 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:21.149994 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.149954 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.150150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.149999 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.150150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.150013 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.150150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.150027 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.150150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.150040 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p48bs\" (UniqueName: \"kubernetes.io/projected/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-kube-api-access-p48bs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.150150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.150054 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.150150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.150066 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.251166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.251122 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-openshift-ca\") pod \"6c0bc38b-efce-493f-aa8b-4c4c28181384\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " Oct 31 13:37:21.251354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.251236 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-xks-ca\") pod \"6c0bc38b-efce-493f-aa8b-4c4c28181384\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " Oct 31 13:37:21.251354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.251310 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0bc38b-efce-493f-aa8b-4c4c28181384-kube-api-access-wlw42\" (UniqueName: \"kubernetes.io/projected/6c0bc38b-efce-493f-aa8b-4c4c28181384-kube-api-access-wlw42\") pod \"6c0bc38b-efce-493f-aa8b-4c4c28181384\" (UID: \"6c0bc38b-efce-493f-aa8b-4c4c28181384\") " Oct 31 13:37:21.251491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.251468 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-openshift-ca" pod "6c0bc38b-efce-493f-aa8b-4c4c28181384" (UID: "6c0bc38b-efce-493f-aa8b-4c4c28181384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:21.252330 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.252296 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.254114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.254086 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0bc38b-efce-493f-aa8b-4c4c28181384-kube-api-access-wlw42" pod "6c0bc38b-efce-493f-aa8b-4c4c28181384" (UID: "6c0bc38b-efce-493f-aa8b-4c4c28181384"). InnerVolumeSpecName "kube-api-access-wlw42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:21.254453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.254417 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-xks-ca" pod "6c0bc38b-efce-493f-aa8b-4c4c28181384" (UID: "6c0bc38b-efce-493f-aa8b-4c4c28181384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:21.353968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.353929 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0bc38b-efce-493f-aa8b-4c4c28181384-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.353968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.353965 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlw42\" (UniqueName: \"kubernetes.io/projected/6c0bc38b-efce-493f-aa8b-4c4c28181384-kube-api-access-wlw42\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:21.826815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.826785 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" Oct 31 13:37:21.826815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.826798 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v" event={"ID":"330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf","Type":"ContainerDied","Data":"ccbc1ee76658d3225b5593756821f061a6f7effa49dd0606f1d0bec131d563f1"} Oct 31 13:37:21.827326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.826908 5074 scope.go:122] "RemoveContainer" containerID="aaaf8c08c1d00e8c8e89baa3b6cb1f1f870cd6bd9c5ca866cfbf7bc468ab3191" Oct 31 13:37:21.828396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.828376 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" Oct 31 13:37:21.828488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.828424 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx" event={"ID":"6c0bc38b-efce-493f-aa8b-4c4c28181384","Type":"ContainerDied","Data":"d4c40972ff686c934ec3df913e5334849620d3991d9f09702484cbf3a153160f"} Oct 31 13:37:21.828488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.828483 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4c40972ff686c934ec3df913e5334849620d3991d9f09702484cbf3a153160f" Oct 31 13:37:21.857682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.857653 5074 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 13:37:21.857827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.857701 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 13:37:21.906223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.906184 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v"] Oct 31 13:37:21.911554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.911511 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0c1fbb-openshift-default-74f567c889-zpw5v"] Oct 31 13:37:21.922955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.922925 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn"] Oct 31 13:37:21.923184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.923152 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" containerID="cri-o://2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0" gracePeriod=30 Oct 31 13:37:21.929220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.929195 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl"] Oct 31 13:37:21.930133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.930097 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" containerID="cri-o://8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e" gracePeriod=30 Oct 31 13:37:21.942458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.942423 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" path="/var/lib/kubelet/pods/330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf/volumes" Oct 31 13:37:21.943287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:21.943260 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" path="/var/lib/kubelet/pods/c18c2078-ee42-451c-9685-1e6f118ac9ec/volumes" Oct 31 13:37:22.857843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:22.857805 5074 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 13:37:22.858275 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:22.857865 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 13:37:23.748870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:23.748787 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" podUID="97ec7097-1544-40c6-92c0-14c712036295" containerName="payload-pre-processing" containerID="cri-o://47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64" gracePeriod=25 Oct 31 13:37:23.858782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:23.858745 5074 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 13:37:23.859267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:23.858806 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 13:37:24.857791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:24.857758 5074 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 13:37:24.857998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:24.857817 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 13:37:25.857853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:25.857812 5074 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" start-of-body= Oct 31 13:37:25.858291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:25.857870 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.89:15021/healthz/ready\": dial tcp 10.132.1.89:15021: connect: connection refused" Oct 31 13:37:26.050597 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:26.050558 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:37:26.863246 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:26.863217 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:26.863787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:26.863741 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:26.864569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:26.864543 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:37:27.047123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.047090 5074 patch_prober.go:29] interesting pod/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:37:27.047281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.047153 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:37:27.328015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.327896 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:37:27.417962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.417929 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:37:27.450052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450020 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istiod-ca-cert\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450065 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-credential-socket\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.450208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450099 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-data\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.450208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450196 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-socket\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.450392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450224 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-token\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-token\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450286 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-data\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450361 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-podinfo\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.450565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450390 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-envoy\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.450565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450441 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istiod-ca-cert" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:27.450565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450469 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-podinfo\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450550 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-envoy\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450606 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-socket" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.450850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450656 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-credential-socket\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450725 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-socket\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450734 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-data" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.450850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450753 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-certs\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.450850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450816 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-certs\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.451192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450896 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-token\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-token\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.451192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450918 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istiod-ca-cert\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.451192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450928 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-certs" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.451192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.450945 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-kube-api-access-xp669\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-kube-api-access-xp669\") pod \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\" (UID: \"998eeb9a-fec6-49bc-8657-c88bf5eecdef\") " Oct 31 13:37:27.451192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.451020 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-data" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.451192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.451115 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-kube-api-access-dlqn2\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-kube-api-access-dlqn2\") pod \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\" (UID: \"90e181b9-7438-4dcb-9cab-ff8bac239f0e\") " Oct 31 13:37:27.452353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.451989 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.452353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.452020 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.452353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.452037 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.452353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.452051 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.452353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.452066 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.452886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.452861 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-token" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:27.453108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.453090 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-credential-socket" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.453345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.453327 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istiod-ca-cert" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:27.453548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.453530 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-certs" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.454088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.454062 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-podinfo" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:27.454214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.454194 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-kube-api-access-dlqn2" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "kube-api-access-dlqn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:27.454323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.454302 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-podinfo" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:27.454374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.454309 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-envoy" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.454374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.454350 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-socket" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.454758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.454738 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-credential-socket" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.455583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.455556 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-envoy" pod "90e181b9-7438-4dcb-9cab-ff8bac239f0e" (UID: "90e181b9-7438-4dcb-9cab-ff8bac239f0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:27.455583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.455571 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-kube-api-access-xp669" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "kube-api-access-xp669". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:27.456069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.456049 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-token" pod "998eeb9a-fec6-49bc-8657-c88bf5eecdef" (UID: "998eeb9a-fec6-49bc-8657-c88bf5eecdef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559024 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559072 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559088 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559105 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559121 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559137 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e181b9-7438-4dcb-9cab-ff8bac239f0e-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559151 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559164 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559177 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998eeb9a-fec6-49bc-8657-c88bf5eecdef-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559203 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp669\" (UniqueName: \"kubernetes.io/projected/998eeb9a-fec6-49bc-8657-c88bf5eecdef-kube-api-access-xp669\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559218 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqn2\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-kube-api-access-dlqn2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559234 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998eeb9a-fec6-49bc-8657-c88bf5eecdef-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.559569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.559254 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e181b9-7438-4dcb-9cab-ff8bac239f0e-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:27.883837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.883796 5074 generic.go:356] "Generic (PLEG): container finished" podID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerID="8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e" exitCode=0 Oct 31 13:37:27.884317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.883841 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" Oct 31 13:37:27.884317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.883883 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" event={"ID":"998eeb9a-fec6-49bc-8657-c88bf5eecdef","Type":"ContainerDied","Data":"8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e"} Oct 31 13:37:27.884317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.883945 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl" event={"ID":"998eeb9a-fec6-49bc-8657-c88bf5eecdef","Type":"ContainerDied","Data":"931415f934e4ca70df69e090804b0b65e089aa88beb4015bfb3ca612333c92bf"} Oct 31 13:37:27.884317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.883969 5074 scope.go:122] "RemoveContainer" containerID="8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e" Oct 31 13:37:27.886732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.886152 5074 generic.go:356] "Generic (PLEG): container finished" podID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerID="2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0" exitCode=0 Oct 31 13:37:27.886732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.886215 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" Oct 31 13:37:27.887476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.887001 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" event={"ID":"90e181b9-7438-4dcb-9cab-ff8bac239f0e","Type":"ContainerDied","Data":"2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0"} Oct 31 13:37:27.887476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.887055 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn" event={"ID":"90e181b9-7438-4dcb-9cab-ff8bac239f0e","Type":"ContainerDied","Data":"35e50765f3f271ec1ef234315163e87c92f777c1f03b1bc755b541b2dcc13760"} Oct 31 13:37:27.887720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.887504 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" podUID="97ec7097-1544-40c6-92c0-14c712036295" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:37:27.903492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.903471 5074 scope.go:122] "RemoveContainer" containerID="8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e" Oct 31 13:37:27.903929 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:27.903901 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e\": container with ID starting with 8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e not found: ID does not exist" containerID="8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e" Oct 31 13:37:27.904028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.903936 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e"} err="failed to get container status \"8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e\": rpc error: code = NotFound desc = could not find container \"8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e\": container with ID starting with 8bf39340703d2014a30d6704f79df38bbcf91c70c6b54dd7661d6724f0cbce5e not found: ID does not exist" Oct 31 13:37:27.904028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.903952 5074 scope.go:122] "RemoveContainer" containerID="2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0" Oct 31 13:37:27.917767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.917744 5074 scope.go:122] "RemoveContainer" containerID="2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0" Oct 31 13:37:27.918073 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:27.918049 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0\": container with ID starting with 2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0 not found: ID does not exist" containerID="2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0" Oct 31 13:37:27.918151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.918080 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0"} err="failed to get container status \"2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0\": rpc error: code = NotFound desc = could not find container \"2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0\": container with ID starting with 2f99229622870e86542156de108d19e0d388fbd9590a188aef65f115c5ad72a0 not found: ID does not exist" Oct 31 13:37:27.922086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.922062 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl"] Oct 31 13:37:27.926557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.926533 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c2c98b-openshift-default-77546954d5-65khl"] Oct 31 13:37:27.943943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.943914 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" path="/var/lib/kubelet/pods/998eeb9a-fec6-49bc-8657-c88bf5eecdef/volumes" Oct 31 13:37:27.944290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.944274 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn"] Oct 31 13:37:27.944342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:27.944305 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-15a6a2-openshift-default-846d8f45b9-x22wn"] Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.468909 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd"] Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470751 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6c0bc38b-efce-493f-aa8b-4c4c28181384" containerName="revoke-keys" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470777 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0bc38b-efce-493f-aa8b-4c4c28181384" containerName="revoke-keys" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470810 5074 cpu_manager.go:414] "containerMap: removing container" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470820 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470867 5074 cpu_manager.go:414] "containerMap: removing container" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470877 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470935 5074 cpu_manager.go:414] "containerMap: removing container" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.470943 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471012 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471026 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471039 5074 cpu_manager.go:414] "containerMap: removing container" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471046 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471405 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="330a70c4-05f0-4716-b8e4-9bdd9a1d1fdf" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471445 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18c2078-ee42-451c-9685-1e6f118ac9ec" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471455 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="998eeb9a-fec6-49bc-8657-c88bf5eecdef" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471476 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0bc38b-efce-493f-aa8b-4c4c28181384" containerName="revoke-keys" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471494 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" containerName="istio-proxy" Oct 31 13:37:28.472334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.471512 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="391f2b95-3fd5-41cb-867d-74f0bfa8cc2e" containerName="istio-proxy" Oct 31 13:37:28.504210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.494364 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.504210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.494400 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd"] Oct 31 13:37:28.504210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.499432 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:37:28.504210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.500052 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:37:28.597719 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.597671 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/96340e4f-525f-4547-8f4a-64e1ad317fbb-kube-api-access-pxqsw\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.597901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.597866 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.598122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.598096 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.700107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.700066 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.700289 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.700200 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/96340e4f-525f-4547-8f4a-64e1ad317fbb-kube-api-access-pxqsw\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.700378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.700327 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.700888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.700867 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.703008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.702978 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.713127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.713094 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/96340e4f-525f-4547-8f4a-64e1ad317fbb-kube-api-access-pxqsw\") pod \"maas-api-revoke-sub-6a9de603-wqvpd\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:28.813971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:28.813881 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:29.105312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:29.105231 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd"] Oct 31 13:37:29.112060 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:37:29.111621 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96340e4f_525f_4547_8f4a_64e1ad317fbb.slice/crio-df008a849c6ae0d838205ea24f765259985bcb0554187e201ec86e51d9ae1265 WatchSource:0}: Error finding container df008a849c6ae0d838205ea24f765259985bcb0554187e201ec86e51d9ae1265: Status 404 returned error can't find the container with id df008a849c6ae0d838205ea24f765259985bcb0554187e201ec86e51d9ae1265 Oct 31 13:37:29.924553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:29.924517 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" event={"ID":"96340e4f-525f-4547-8f4a-64e1ad317fbb","Type":"ContainerStarted","Data":"94d3da0192260b7e2d840e20647446c1109329808e2dad6481754a070a3654fb"} Oct 31 13:37:29.924784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:29.924586 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" event={"ID":"96340e4f-525f-4547-8f4a-64e1ad317fbb","Type":"ContainerStarted","Data":"df008a849c6ae0d838205ea24f765259985bcb0554187e201ec86e51d9ae1265"} Oct 31 13:37:29.940080 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:29.940051 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e181b9-7438-4dcb-9cab-ff8bac239f0e" path="/var/lib/kubelet/pods/90e181b9-7438-4dcb-9cab-ff8bac239f0e/volumes" Oct 31 13:37:29.969431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:29.969365 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" podStartSLOduration=1.969347256 podStartE2EDuration="1.969347256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:29.966299647 +0000 UTC m=+3944.144017630" watchObservedRunningTime="2026-10-31 13:37:29.969347256 +0000 UTC m=+3944.147065242" Oct 31 13:37:31.012691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:31.012625 5074 generic.go:356] "Generic (PLEG): container finished" podID="96340e4f-525f-4547-8f4a-64e1ad317fbb" containerID="94d3da0192260b7e2d840e20647446c1109329808e2dad6481754a070a3654fb" exitCode=0 Oct 31 13:37:31.014190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:31.014154 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" event={"ID":"96340e4f-525f-4547-8f4a-64e1ad317fbb","Type":"ContainerDied","Data":"94d3da0192260b7e2d840e20647446c1109329808e2dad6481754a070a3654fb"} Oct 31 13:37:32.508396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.508023 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:32.627497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.627306 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-openshift-ca\") pod \"96340e4f-525f-4547-8f4a-64e1ad317fbb\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " Oct 31 13:37:32.628041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.628009 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96340e4f-525f-4547-8f4a-64e1ad317fbb-kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/96340e4f-525f-4547-8f4a-64e1ad317fbb-kube-api-access-pxqsw\") pod \"96340e4f-525f-4547-8f4a-64e1ad317fbb\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " Oct 31 13:37:32.628368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.628350 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-xks-ca\") pod \"96340e4f-525f-4547-8f4a-64e1ad317fbb\" (UID: \"96340e4f-525f-4547-8f4a-64e1ad317fbb\") " Oct 31 13:37:32.631240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.631215 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-openshift-ca" pod "96340e4f-525f-4547-8f4a-64e1ad317fbb" (UID: "96340e4f-525f-4547-8f4a-64e1ad317fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:32.688915 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.688830 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96340e4f-525f-4547-8f4a-64e1ad317fbb-kube-api-access-pxqsw" pod "96340e4f-525f-4547-8f4a-64e1ad317fbb" (UID: "96340e4f-525f-4547-8f4a-64e1ad317fbb"). InnerVolumeSpecName "kube-api-access-pxqsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:32.689668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.689604 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-xks-ca" pod "96340e4f-525f-4547-8f4a-64e1ad317fbb" (UID: "96340e4f-525f-4547-8f4a-64e1ad317fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:32.741338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.741295 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:32.741580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.741561 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96340e4f-525f-4547-8f4a-64e1ad317fbb-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:32.741723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:32.741709 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqsw\" (UniqueName: \"kubernetes.io/projected/96340e4f-525f-4547-8f4a-64e1ad317fbb-kube-api-access-pxqsw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:33.038709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:33.038094 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" event={"ID":"96340e4f-525f-4547-8f4a-64e1ad317fbb","Type":"ContainerDied","Data":"df008a849c6ae0d838205ea24f765259985bcb0554187e201ec86e51d9ae1265"} Oct 31 13:37:33.038709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:33.038143 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df008a849c6ae0d838205ea24f765259985bcb0554187e201ec86e51d9ae1265" Oct 31 13:37:33.038709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:33.038195 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd" Oct 31 13:37:34.931656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:34.931225 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 13:37:35.059657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:35.059297 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:35.111759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:35.111714 5074 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:35.134653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:35.133737 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:35.138664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:35.134978 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 13:37:35.229653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:35.220806 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:35.283818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:35.283767 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:36.006665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.005763 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:36.007165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.006672 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 13:37:36.198705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.144800 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:36.198705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.145392 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:37:36.198705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.189764 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:36.198705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.190525 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 13:37:36.303706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.260526 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:36.303706 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.281754 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:36.311671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.308509 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 13:37:36.311671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.308929 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 13:37:36.311671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.310677 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:36.311671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.311000 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 13:37:36.805867 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.805830 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 13:37:36.807095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.807070 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 13:37:36.961699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:36.960304 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:37:37.005866 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:37.005834 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 13:37:37.031436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:37.031401 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 13:37:37.910657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:37.909263 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" podUID="97ec7097-1544-40c6-92c0-14c712036295" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:37:43.624880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:43.624837 5074 generic.go:356] "Generic (PLEG): container finished" podID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerID="df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca" exitCode=0 Oct 31 13:37:43.625909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:43.625530 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" event={"ID":"78c86405-b843-4803-8b7a-2698ce6dc7be","Type":"ContainerDied","Data":"df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca"} Oct 31 13:37:43.626762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:43.626743 5074 scope.go:122] "RemoveContainer" containerID="df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca" Oct 31 13:37:43.642619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:43.642428 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:43.644325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:43.644296 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:43.644813 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:43.644790 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:44.091006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.090978 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:37:44.212822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.212736 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ec7097-1544-40c6-92c0-14c712036295-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec7097-1544-40c6-92c0-14c712036295-plugins-config-volume\") pod \"97ec7097-1544-40c6-92c0-14c712036295\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " Oct 31 13:37:44.213011 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.212825 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ec7097-1544-40c6-92c0-14c712036295-kube-api-access-wcb4d\" (UniqueName: \"kubernetes.io/projected/97ec7097-1544-40c6-92c0-14c712036295-kube-api-access-wcb4d\") pod \"97ec7097-1544-40c6-92c0-14c712036295\" (UID: \"97ec7097-1544-40c6-92c0-14c712036295\") " Oct 31 13:37:44.213223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.213195 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ec7097-1544-40c6-92c0-14c712036295-plugins-config-volume" pod "97ec7097-1544-40c6-92c0-14c712036295" (UID: "97ec7097-1544-40c6-92c0-14c712036295"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:44.213708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.213672 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ec7097-1544-40c6-92c0-14c712036295-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:44.215776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.215743 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ec7097-1544-40c6-92c0-14c712036295-kube-api-access-wcb4d" pod "97ec7097-1544-40c6-92c0-14c712036295" (UID: "97ec7097-1544-40c6-92c0-14c712036295"). InnerVolumeSpecName "kube-api-access-wcb4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:44.315254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.315219 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcb4d\" (UniqueName: \"kubernetes.io/projected/97ec7097-1544-40c6-92c0-14c712036295-kube-api-access-wcb4d\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:44.634831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.634792 5074 generic.go:356] "Generic (PLEG): container finished" podID="97ec7097-1544-40c6-92c0-14c712036295" containerID="47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64" exitCode=0 Oct 31 13:37:44.635279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.634842 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" Oct 31 13:37:44.635279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.634878 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" event={"ID":"97ec7097-1544-40c6-92c0-14c712036295","Type":"ContainerDied","Data":"47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64"} Oct 31 13:37:44.635279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.634938 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6" event={"ID":"97ec7097-1544-40c6-92c0-14c712036295","Type":"ContainerDied","Data":"ac58c30551a0cd93c31fb2d77b35b525c1b2f1b55441b4f1f76f5ad9fd489884"} Oct 31 13:37:44.635279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.634961 5074 scope.go:122] "RemoveContainer" containerID="47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64" Oct 31 13:37:44.637203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.637178 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" event={"ID":"78c86405-b843-4803-8b7a-2698ce6dc7be","Type":"ContainerStarted","Data":"f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208"} Oct 31 13:37:44.647021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.646849 5074 scope.go:122] "RemoveContainer" containerID="47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64" Oct 31 13:37:44.647142 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:44.647120 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64\": container with ID starting with 47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64 not found: ID does not exist" containerID="47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64" Oct 31 13:37:44.647179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.647152 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64"} err="failed to get container status \"47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64\": rpc error: code = NotFound desc = could not find container \"47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64\": container with ID starting with 47ef4bbc54955f3ad06bc4fcec6f8999b6be919743a3f8f33b6904988de62e64 not found: ID does not exist" Oct 31 13:37:44.708519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.708481 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6"] Oct 31 13:37:44.713029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.712996 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cddb3-75dcc4d99-76kb6"] Oct 31 13:37:44.744175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.744147 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:37:44.745762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.745733 5074 patch_prober.go:29] interesting pod/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 13:37:44.745879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:44.745793 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 13:37:45.737403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:45.737365 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:37:45.744370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:45.744341 5074 patch_prober.go:29] interesting pod/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 13:37:45.744522 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:45.744400 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 13:37:45.942113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:45.942080 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ec7097-1544-40c6-92c0-14c712036295" path="/var/lib/kubelet/pods/97ec7097-1544-40c6-92c0-14c712036295/volumes" Oct 31 13:37:46.744105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:46.744066 5074 patch_prober.go:29] interesting pod/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 13:37:46.744499 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:46.744133 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 13:37:47.743807 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:47.743771 5074 patch_prober.go:29] interesting pod/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 13:37:47.743988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:47.743827 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 13:37:48.743787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:48.743738 5074 patch_prober.go:29] interesting pod/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 13:37:48.744281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:48.743823 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 13:37:49.422005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:49.421958 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr"] Oct 31 13:37:49.765012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:49.758118 5074 patch_prober.go:29] interesting pod/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 13:37:49.765012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:49.758178 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 13:37:50.748982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:50.748944 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:37:50.750351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:50.750321 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:37:54.422560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:54.422518 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" podUID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" containerName="payload-pre-processing" containerID="cri-o://7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261" gracePeriod=25 Oct 31 13:37:55.863600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:37:55.863560 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" podUID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:37:55.985696 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:37:55.985658 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:38:05.863360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:05.863318 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" podUID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:38:09.322367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.322327 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924"] Oct 31 13:38:09.324157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.324130 5074 cpu_manager.go:414] "containerMap: removing container" podUID="97ec7097-1544-40c6-92c0-14c712036295" containerName="payload-pre-processing" Oct 31 13:38:09.324359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.324344 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ec7097-1544-40c6-92c0-14c712036295" containerName="payload-pre-processing" Oct 31 13:38:09.324503 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.324478 5074 cpu_manager.go:414] "containerMap: removing container" podUID="96340e4f-525f-4547-8f4a-64e1ad317fbb" containerName="revoke-keys" Oct 31 13:38:09.324600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.324588 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96340e4f-525f-4547-8f4a-64e1ad317fbb" containerName="revoke-keys" Oct 31 13:38:09.325076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.325021 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="96340e4f-525f-4547-8f4a-64e1ad317fbb" containerName="revoke-keys" Oct 31 13:38:09.325237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.325225 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ec7097-1544-40c6-92c0-14c712036295" containerName="payload-pre-processing" Oct 31 13:38:09.333355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.333331 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.358738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.358706 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:38:09.359132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.359111 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:38:09.379663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.377878 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924"] Oct 31 13:38:09.415465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.415427 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.418313 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.418210 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.418497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.418404 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2hf\" (UniqueName: \"kubernetes.io/projected/07ba542c-566c-4c81-970f-969d0a6e7153-kube-api-access-bn2hf\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.526396 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.526361 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2hf\" (UniqueName: \"kubernetes.io/projected/07ba542c-566c-4c81-970f-969d0a6e7153-kube-api-access-bn2hf\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.526581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.526502 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.526581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.526543 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.527255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.527226 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.528896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.528871 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.541375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.541343 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2hf\" (UniqueName: \"kubernetes.io/projected/07ba542c-566c-4c81-970f-969d0a6e7153-kube-api-access-bn2hf\") pod \"maas-api-revoke-sub-3e34628e-98924\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:09.703033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:09.702938 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:10.003857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:10.003822 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924"] Oct 31 13:38:10.004886 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:38:10.004855 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ba542c_566c_4c81_970f_969d0a6e7153.slice/crio-0ed012639ef0b8e6d99a77793f5b2ddd140e088f36834545130386d16f2e9f8b WatchSource:0}: Error finding container 0ed012639ef0b8e6d99a77793f5b2ddd140e088f36834545130386d16f2e9f8b: Status 404 returned error can't find the container with id 0ed012639ef0b8e6d99a77793f5b2ddd140e088f36834545130386d16f2e9f8b Oct 31 13:38:10.110024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:10.109972 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:38:10.986337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:10.986311 5074 generic.go:356] "Generic (PLEG): container finished" podID="07ba542c-566c-4c81-970f-969d0a6e7153" containerID="493cd6121b8f8bf9380ffe15576c3eb7944986dc44768dabb07aa1704f543371" exitCode=0 Oct 31 13:38:10.986681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:10.986522 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" event={"ID":"07ba542c-566c-4c81-970f-969d0a6e7153","Type":"ContainerDied","Data":"493cd6121b8f8bf9380ffe15576c3eb7944986dc44768dabb07aa1704f543371"} Oct 31 13:38:10.986681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:10.986563 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" event={"ID":"07ba542c-566c-4c81-970f-969d0a6e7153","Type":"ContainerStarted","Data":"0ed012639ef0b8e6d99a77793f5b2ddd140e088f36834545130386d16f2e9f8b"} Oct 31 13:38:12.175418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.175384 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:12.276482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.276394 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ba542c-566c-4c81-970f-969d0a6e7153-kube-api-access-bn2hf\" (UniqueName: \"kubernetes.io/projected/07ba542c-566c-4c81-970f-969d0a6e7153-kube-api-access-bn2hf\") pod \"07ba542c-566c-4c81-970f-969d0a6e7153\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " Oct 31 13:38:12.276713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.276674 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-openshift-ca\") pod \"07ba542c-566c-4c81-970f-969d0a6e7153\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " Oct 31 13:38:12.276792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.276771 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-xks-ca\") pod \"07ba542c-566c-4c81-970f-969d0a6e7153\" (UID: \"07ba542c-566c-4c81-970f-969d0a6e7153\") " Oct 31 13:38:12.277055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.277031 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-openshift-ca" pod "07ba542c-566c-4c81-970f-969d0a6e7153" (UID: "07ba542c-566c-4c81-970f-969d0a6e7153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:12.277343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.277325 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:12.278803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.278770 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ba542c-566c-4c81-970f-969d0a6e7153-kube-api-access-bn2hf" pod "07ba542c-566c-4c81-970f-969d0a6e7153" (UID: "07ba542c-566c-4c81-970f-969d0a6e7153"). InnerVolumeSpecName "kube-api-access-bn2hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:12.278924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.278810 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-xks-ca" pod "07ba542c-566c-4c81-970f-969d0a6e7153" (UID: "07ba542c-566c-4c81-970f-969d0a6e7153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:12.378651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.378609 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ba542c-566c-4c81-970f-969d0a6e7153-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:12.378815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:12.378658 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2hf\" (UniqueName: \"kubernetes.io/projected/07ba542c-566c-4c81-970f-969d0a6e7153-kube-api-access-bn2hf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:13.002726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:13.002693 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" Oct 31 13:38:13.002924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:13.002692 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924" event={"ID":"07ba542c-566c-4c81-970f-969d0a6e7153","Type":"ContainerDied","Data":"0ed012639ef0b8e6d99a77793f5b2ddd140e088f36834545130386d16f2e9f8b"} Oct 31 13:38:13.002924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:13.002831 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ed012639ef0b8e6d99a77793f5b2ddd140e088f36834545130386d16f2e9f8b" Oct 31 13:38:14.746149 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:14.746113 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:38:14.922756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:14.916694 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-kube-api-access-zflbx\" (UniqueName: \"kubernetes.io/projected/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-kube-api-access-zflbx\") pod \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " Oct 31 13:38:14.922756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:14.916795 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-plugins-config-volume\") pod \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\" (UID: \"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04\") " Oct 31 13:38:14.922756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:14.918548 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-plugins-config-volume" pod "e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" (UID: "e149ec92-34f0-4b3a-bb1a-fd851cdc8b04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:14.922756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:14.921137 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-kube-api-access-zflbx" pod "e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" (UID: "e149ec92-34f0-4b3a-bb1a-fd851cdc8b04"). InnerVolumeSpecName "kube-api-access-zflbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:15.019682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.019625 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zflbx\" (UniqueName: \"kubernetes.io/projected/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-kube-api-access-zflbx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:15.019682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.019679 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:15.062151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.062118 5074 generic.go:356] "Generic (PLEG): container finished" podID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" containerID="7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261" exitCode=0 Oct 31 13:38:15.062326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.062168 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" Oct 31 13:38:15.062326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.062205 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" event={"ID":"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04","Type":"ContainerDied","Data":"7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261"} Oct 31 13:38:15.062326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.062265 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr" event={"ID":"e149ec92-34f0-4b3a-bb1a-fd851cdc8b04","Type":"ContainerDied","Data":"bb1b67d5b169cad4f4da05191a46e2e6c42e7b5daa6d360e799bf51644ebc9c4"} Oct 31 13:38:15.062326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.062288 5074 scope.go:122] "RemoveContainer" containerID="7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261" Oct 31 13:38:15.075441 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.075422 5074 scope.go:122] "RemoveContainer" containerID="7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261" Oct 31 13:38:15.075774 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:15.075755 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261\": container with ID starting with 7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261 not found: ID does not exist" containerID="7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261" Oct 31 13:38:15.075839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.075783 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261"} err="failed to get container status \"7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261\": rpc error: code = NotFound desc = could not find container \"7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261\": container with ID starting with 7e9ea97521b52091b467f287eb661252b9706d10a6cb2e020040f5cf3a9e7261 not found: ID does not exist" Oct 31 13:38:15.090491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.090456 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr"] Oct 31 13:38:15.094658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.094619 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09db67-79c685cc5f-kh5fr"] Oct 31 13:38:15.941267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:15.941223 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" path="/var/lib/kubelet/pods/e149ec92-34f0-4b3a-bb1a-fd851cdc8b04/volumes" Oct 31 13:38:17.748508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:17.748466 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7"] Oct 31 13:38:17.749116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:17.748761 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" containerID="cri-o://1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799" gracePeriod=30 Oct 31 13:38:23.060846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.060806 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:38:23.165274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165240 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-credential-socket\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165400 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-token\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-token\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165451 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-podinfo\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165528 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-certs\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165546 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-credential-socket" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:23.165600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165588 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-envoy\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165652 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-data\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165691 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa965c2-ad6a-4339-879f-41318554f6ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa965c2-ad6a-4339-879f-41318554f6ce-istiod-ca-cert\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165780 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-socket\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.165974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.165852 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-kube-api-access-dbwvp\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-kube-api-access-dbwvp\") pod \"6aa965c2-ad6a-4339-879f-41318554f6ce\" (UID: \"6aa965c2-ad6a-4339-879f-41318554f6ce\") " Oct 31 13:38:23.166172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.166144 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-data" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:23.166646 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.166611 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.166758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.166649 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.167220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.166838 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-socket" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:23.167329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.166919 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-certs" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:23.167329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.167176 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa965c2-ad6a-4339-879f-41318554f6ce-istiod-ca-cert" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:23.167929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.167901 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-token" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:23.168031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.168005 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-podinfo" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:38:23.168223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.168204 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-kube-api-access-dbwvp" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "kube-api-access-dbwvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:23.168756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.168732 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-envoy" pod "6aa965c2-ad6a-4339-879f-41318554f6ce" (UID: "6aa965c2-ad6a-4339-879f-41318554f6ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:23.219363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.219322 5074 generic.go:356] "Generic (PLEG): container finished" podID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerID="1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799" exitCode=0 Oct 31 13:38:23.219548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.219370 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" event={"ID":"6aa965c2-ad6a-4339-879f-41318554f6ce","Type":"ContainerDied","Data":"1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799"} Oct 31 13:38:23.219548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.219442 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" event={"ID":"6aa965c2-ad6a-4339-879f-41318554f6ce","Type":"ContainerDied","Data":"5f77cdd761d9a523adca05c1f0fa95fdba9bc11671864577ff3025d3ebcbe431"} Oct 31 13:38:23.219548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.219473 5074 scope.go:122] "RemoveContainer" containerID="1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799" Oct 31 13:38:23.219729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.219673 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7" Oct 31 13:38:23.232617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.232597 5074 scope.go:122] "RemoveContainer" containerID="392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc" Oct 31 13:38:23.242902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.242883 5074 scope.go:122] "RemoveContainer" containerID="1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799" Oct 31 13:38:23.243181 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:23.243156 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799\": container with ID starting with 1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799 not found: ID does not exist" containerID="1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799" Oct 31 13:38:23.243283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.243187 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799"} err="failed to get container status \"1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799\": rpc error: code = NotFound desc = could not find container \"1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799\": container with ID starting with 1b5aa8f8b05b71cef3feccbe8b0c28a85f74413b18bd72df636343eecc526799 not found: ID does not exist" Oct 31 13:38:23.243283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.243204 5074 scope.go:122] "RemoveContainer" containerID="392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc" Oct 31 13:38:23.243437 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:23.243419 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc\": container with ID starting with 392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc not found: ID does not exist" containerID="392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc" Oct 31 13:38:23.243487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.243440 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc"} err="failed to get container status \"392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc\": rpc error: code = NotFound desc = could not find container \"392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc\": container with ID starting with 392155449bca3d437cd0595981cd3e10429313ed9454ce36b3eeae5ac719a4bc not found: ID does not exist" Oct 31 13:38:23.255479 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.255451 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7"] Oct 31 13:38:23.260808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.260782 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-09db67-openshift-default-f57fb858c-v65p7"] Oct 31 13:38:23.268024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.268000 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.268140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.268028 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.268140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.268045 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.268140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.268059 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.268140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.268073 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa965c2-ad6a-4339-879f-41318554f6ce-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.268140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.268087 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa965c2-ad6a-4339-879f-41318554f6ce-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.268140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.268101 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwvp\" (UniqueName: \"kubernetes.io/projected/6aa965c2-ad6a-4339-879f-41318554f6ce-kube-api-access-dbwvp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:23.940406 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:23.940371 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" path="/var/lib/kubelet/pods/6aa965c2-ad6a-4339-879f-41318554f6ce/volumes" Oct 31 13:38:26.121307 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:26.118820 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:38:51.650056 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:51.650007 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62"] Oct 31 13:38:51.666112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:51.666072 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j"] Oct 31 13:38:52.752786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:52.752741 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc"] Oct 31 13:38:52.753257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:52.753042 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" containerID="cri-o://4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc" gracePeriod=30 Oct 31 13:38:55.639924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:55.639884 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6"] Oct 31 13:38:55.640496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:55.640160 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" podUID="f4d7431e-9744-41b5-9c3e-25d4bdd552fa" containerName="maas-api" containerID="cri-o://54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418" gracePeriod=30 Oct 31 13:38:56.266823 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:56.265774 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:38:56.316542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.316508 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:38:56.369970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.364173 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-openshift-service-ca\") pod \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " Oct 31 13:38:56.369970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.364246 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-maas-api-tls\") pod \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " Oct 31 13:38:56.369970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.364542 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-metrics-certs\") pod \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " Oct 31 13:38:56.369970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.364599 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-kube-api-access-6bqkn\" (UniqueName: \"kubernetes.io/projected/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-kube-api-access-6bqkn\") pod \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\" (UID: \"f4d7431e-9744-41b5-9c3e-25d4bdd552fa\") " Oct 31 13:38:56.392738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.371665 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-openshift-service-ca" pod "f4d7431e-9744-41b5-9c3e-25d4bdd552fa" (UID: "f4d7431e-9744-41b5-9c3e-25d4bdd552fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:56.482692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.469121 5074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-openshift-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:56.581960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.581408 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-metrics-certs" pod "f4d7431e-9744-41b5-9c3e-25d4bdd552fa" (UID: "f4d7431e-9744-41b5-9c3e-25d4bdd552fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:56.581960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.581797 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-kube-api-access-6bqkn" pod "f4d7431e-9744-41b5-9c3e-25d4bdd552fa" (UID: "f4d7431e-9744-41b5-9c3e-25d4bdd552fa"). InnerVolumeSpecName "kube-api-access-6bqkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:56.583673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.583581 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-maas-api-tls" pod "f4d7431e-9744-41b5-9c3e-25d4bdd552fa" (UID: "f4d7431e-9744-41b5-9c3e-25d4bdd552fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:56.650392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.650342 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" podUID="d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" containerName="payload-processing" containerID="cri-o://4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e" gracePeriod=25 Oct 31 13:38:56.666595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.666526 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" podUID="7af7844c-8037-47ed-8593-84a92852c243" containerName="payload-pre-processing" containerID="cri-o://8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee" gracePeriod=25 Oct 31 13:38:56.674117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.673574 5074 generic.go:356] "Generic (PLEG): container finished" podID="f4d7431e-9744-41b5-9c3e-25d4bdd552fa" containerID="54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418" exitCode=0 Oct 31 13:38:56.674117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.673648 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" Oct 31 13:38:56.674117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.673893 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" event={"ID":"f4d7431e-9744-41b5-9c3e-25d4bdd552fa","Type":"ContainerDied","Data":"54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418"} Oct 31 13:38:56.674117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.673946 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6" event={"ID":"f4d7431e-9744-41b5-9c3e-25d4bdd552fa","Type":"ContainerDied","Data":"0362609d49e1489145f98721ba4c7edb4ed6c69bae3c9c89710f48a176af1752"} Oct 31 13:38:56.674117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.673985 5074 scope.go:122] "RemoveContainer" containerID="54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418" Oct 31 13:38:56.682581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.682509 5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-metrics-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:56.682581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.682539 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bqkn\" (UniqueName: \"kubernetes.io/projected/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-kube-api-access-6bqkn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:56.682581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.682556 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4d7431e-9744-41b5-9c3e-25d4bdd552fa-maas-api-tls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:56.826112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.826082 5074 scope.go:122] "RemoveContainer" containerID="54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418" Oct 31 13:38:56.826528 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:56.826498 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418\": container with ID starting with 54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418 not found: ID does not exist" containerID="54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418" Oct 31 13:38:56.826575 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.826543 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418"} err="failed to get container status \"54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418\": rpc error: code = NotFound desc = could not find container \"54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418\": container with ID starting with 54ae44184b537c6f3bceecf1b90371b96d939bcd1568a03ff6c618f168ef4418 not found: ID does not exist" Oct 31 13:38:56.841231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.841111 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6"] Oct 31 13:38:56.845319 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.845290 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e7896150-86457b6d97-wpdm6"] Oct 31 13:38:56.868439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.868404 5074 patch_prober.go:29] interesting pod/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:38:56.868617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:56.868472 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:38:57.024050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.024026 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:38:57.038824 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.038796 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:38:57.095850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.090659 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af7844c-8037-47ed-8593-84a92852c243-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af7844c-8037-47ed-8593-84a92852c243-plugins-config-volume\") pod \"7af7844c-8037-47ed-8593-84a92852c243\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " Oct 31 13:38:57.095850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.090859 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-plugins-config-volume\") pod \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " Oct 31 13:38:57.095850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.090924 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-kube-api-access-52znb\" (UniqueName: \"kubernetes.io/projected/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-kube-api-access-52znb\") pod \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\" (UID: \"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80\") " Oct 31 13:38:57.095850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.091041 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af7844c-8037-47ed-8593-84a92852c243-kube-api-access-jjwpl\" (UniqueName: \"kubernetes.io/projected/7af7844c-8037-47ed-8593-84a92852c243-kube-api-access-jjwpl\") pod \"7af7844c-8037-47ed-8593-84a92852c243\" (UID: \"7af7844c-8037-47ed-8593-84a92852c243\") " Oct 31 13:38:57.099962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.099920 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-plugins-config-volume" pod "d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" (UID: "d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:57.100608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.100582 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af7844c-8037-47ed-8593-84a92852c243-plugins-config-volume" pod "7af7844c-8037-47ed-8593-84a92852c243" (UID: "7af7844c-8037-47ed-8593-84a92852c243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:57.121370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.121328 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-kube-api-access-52znb" pod "d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" (UID: "d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80"). InnerVolumeSpecName "kube-api-access-52znb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:57.121576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.121501 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af7844c-8037-47ed-8593-84a92852c243-kube-api-access-jjwpl" pod "7af7844c-8037-47ed-8593-84a92852c243" (UID: "7af7844c-8037-47ed-8593-84a92852c243"). InnerVolumeSpecName "kube-api-access-jjwpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:57.194249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.194205 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af7844c-8037-47ed-8593-84a92852c243-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:57.194249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.194249 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:57.194455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.194264 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52znb\" (UniqueName: \"kubernetes.io/projected/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80-kube-api-access-52znb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:57.194455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.194278 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjwpl\" (UniqueName: \"kubernetes.io/projected/7af7844c-8037-47ed-8593-84a92852c243-kube-api-access-jjwpl\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:57.684379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.684341 5074 generic.go:356] "Generic (PLEG): container finished" podID="d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" containerID="4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e" exitCode=0 Oct 31 13:38:57.684850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.684412 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" Oct 31 13:38:57.684850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.684418 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" event={"ID":"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80","Type":"ContainerDied","Data":"4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e"} Oct 31 13:38:57.684850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.684544 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62" event={"ID":"d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80","Type":"ContainerDied","Data":"e23704f28ed923fdd1f3ed9d71975bdea87c527e297c212528125964f3e74066"} Oct 31 13:38:57.684850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.684559 5074 scope.go:122] "RemoveContainer" containerID="4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e" Oct 31 13:38:57.689073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.689042 5074 generic.go:356] "Generic (PLEG): container finished" podID="7af7844c-8037-47ed-8593-84a92852c243" containerID="8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee" exitCode=0 Oct 31 13:38:57.689283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.689080 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" event={"ID":"7af7844c-8037-47ed-8593-84a92852c243","Type":"ContainerDied","Data":"8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee"} Oct 31 13:38:57.689283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.689123 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" event={"ID":"7af7844c-8037-47ed-8593-84a92852c243","Type":"ContainerDied","Data":"bd30a67fcccd1ce92c7fc2b0a7b956ec838479904d42375f14abec77a51544f5"} Oct 31 13:38:57.689283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.689203 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j" Oct 31 13:38:57.703863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.703839 5074 scope.go:122] "RemoveContainer" containerID="4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e" Oct 31 13:38:57.704267 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:57.704233 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e\": container with ID starting with 4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e not found: ID does not exist" containerID="4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e" Oct 31 13:38:57.704388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.704278 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e"} err="failed to get container status \"4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e\": rpc error: code = NotFound desc = could not find container \"4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e\": container with ID starting with 4fb7fc012d62abf4879bdc1e9dadba9d81674eb1463ab5b5e228b542e9fabd7e not found: ID does not exist" Oct 31 13:38:57.704388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.704304 5074 scope.go:122] "RemoveContainer" containerID="8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee" Oct 31 13:38:57.727506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.727477 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j"] Oct 31 13:38:57.728201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.728180 5074 scope.go:122] "RemoveContainer" containerID="8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee" Oct 31 13:38:57.728536 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:57.728508 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee\": container with ID starting with 8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee not found: ID does not exist" containerID="8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee" Oct 31 13:38:57.728627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.728548 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee"} err="failed to get container status \"8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee\": rpc error: code = NotFound desc = could not find container \"8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee\": container with ID starting with 8aaf14e807c0f9b8bbeaedf14c18f77b797ae0209e740389e1408831b2d7e0ee not found: ID does not exist" Oct 31 13:38:57.731787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.731767 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e7896150-7f7fffd6dc-l8g9j"] Oct 31 13:38:57.741237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.741211 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62"] Oct 31 13:38:57.753553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.753522 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e7896150-85d85d76df-vtc62"] Oct 31 13:38:57.832662 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:57.831972 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601212db_06e4_49b6_b027_5d37562ca8c2.slice/crio-4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:38:57.944155 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.944070 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af7844c-8037-47ed-8593-84a92852c243" path="/var/lib/kubelet/pods/7af7844c-8037-47ed-8593-84a92852c243/volumes" Oct 31 13:38:57.944588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.944572 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" path="/var/lib/kubelet/pods/d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80/volumes" Oct 31 13:38:57.945045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:57.945030 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d7431e-9744-41b5-9c3e-25d4bdd552fa" path="/var/lib/kubelet/pods/f4d7431e-9744-41b5-9c3e-25d4bdd552fa/volumes" Oct 31 13:38:58.096348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.096312 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:38:58.211850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.211748 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-kube-api-access-l5kf7\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-kube-api-access-l5kf7\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.211850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.211821 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/601212db-06e4-49b6-b027-5d37562ca8c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601212db-06e4-49b6-b027-5d37562ca8c2-istiod-ca-cert\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.211940 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-socket\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.211988 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-data\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212031 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-credential-socket\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212140 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/601212db-06e4-49b6-b027-5d37562ca8c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601212db-06e4-49b6-b027-5d37562ca8c2-istio-podinfo\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212259 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-envoy\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212262 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-data" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.212420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212314 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601212db-06e4-49b6-b027-5d37562ca8c2-istiod-ca-cert" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:58.212420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212335 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-certs\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212390 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-istio-token\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-istio-token\") pod \"601212db-06e4-49b6-b027-5d37562ca8c2\" (UID: \"601212db-06e4-49b6-b027-5d37562ca8c2\") " Oct 31 13:38:58.212595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212417 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-credential-socket" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.212672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.212615 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-certs" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.213127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.213101 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601212db-06e4-49b6-b027-5d37562ca8c2-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.213251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.213130 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.213251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.213145 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.213251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.213158 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.213440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.213419 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-socket" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.215259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.214891 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-envoy" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:58.215259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.215192 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-kube-api-access-l5kf7" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "kube-api-access-l5kf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:58.215453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.215268 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/601212db-06e4-49b6-b027-5d37562ca8c2-istio-podinfo" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:38:58.215453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.215437 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-istio-token" pod "601212db-06e4-49b6-b027-5d37562ca8c2" (UID: "601212db-06e4-49b6-b027-5d37562ca8c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:58.314596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.314551 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.314596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.314593 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.314841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.314611 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5kf7\" (UniqueName: \"kubernetes.io/projected/601212db-06e4-49b6-b027-5d37562ca8c2-kube-api-access-l5kf7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.314841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.314627 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601212db-06e4-49b6-b027-5d37562ca8c2-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.314841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.314657 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601212db-06e4-49b6-b027-5d37562ca8c2-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:58.734596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.734560 5074 generic.go:356] "Generic (PLEG): container finished" podID="601212db-06e4-49b6-b027-5d37562ca8c2" containerID="4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc" exitCode=0 Oct 31 13:38:58.735115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.734602 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" event={"ID":"601212db-06e4-49b6-b027-5d37562ca8c2","Type":"ContainerDied","Data":"4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc"} Oct 31 13:38:58.735115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.734612 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" Oct 31 13:38:58.735115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.734664 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc" event={"ID":"601212db-06e4-49b6-b027-5d37562ca8c2","Type":"ContainerDied","Data":"1761d0da9fda276abb5aace25f540b94d09cac87bb412daed358bc539e5103f8"} Oct 31 13:38:58.735115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.734693 5074 scope.go:122] "RemoveContainer" containerID="4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc" Oct 31 13:38:58.747318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.747299 5074 scope.go:122] "RemoveContainer" containerID="4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc" Oct 31 13:38:58.747610 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:38:58.747588 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc\": container with ID starting with 4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc not found: ID does not exist" containerID="4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc" Oct 31 13:38:58.747714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.747618 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc"} err="failed to get container status \"4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc\": rpc error: code = NotFound desc = could not find container \"4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc\": container with ID starting with 4e3f67a0f95ddc27a0889b3912fdc5fe1b7bab2f9cb0af35dd0d88e1e2dbaefc not found: ID does not exist" Oct 31 13:38:58.762814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.762792 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc"] Oct 31 13:38:58.775338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:58.775308 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-50b4aa-openshift-default-788995bfb7-vtjtc"] Oct 31 13:38:59.940965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:38:59.940929 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" path="/var/lib/kubelet/pods/601212db-06e4-49b6-b027-5d37562ca8c2/volumes" Oct 31 13:39:01.279345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:01.279291 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs"] Oct 31 13:39:01.280078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:01.279609 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" containerID="cri-o://f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208" gracePeriod=30 Oct 31 13:39:05.751957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:05.751905 5074 patch_prober.go:29] interesting pod/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:39:05.752378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:05.751972 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:39:06.653560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.653533 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:39:06.772504 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772468 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-certs\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772521 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-podinfo\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772543 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-envoy\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772596 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-socket\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772760 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-token\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-token\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772831 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-kube-api-access-vm2vp\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-kube-api-access-vm2vp\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772866 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c86405-b843-4803-8b7a-2698ce6dc7be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c86405-b843-4803-8b7a-2698ce6dc7be-istiod-ca-cert\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772899 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-data\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772940 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-socket" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:06.773036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.772994 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-credential-socket\") pod \"78c86405-b843-4803-8b7a-2698ce6dc7be\" (UID: \"78c86405-b843-4803-8b7a-2698ce6dc7be\") " Oct 31 13:39:06.773523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773153 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-certs" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:06.773523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773372 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-data" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:06.773523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773405 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c86405-b843-4803-8b7a-2698ce6dc7be-istiod-ca-cert" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:39:06.773704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773595 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-credential-socket" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:06.773792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773769 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.773858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773797 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.773858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773811 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.773858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773825 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c86405-b843-4803-8b7a-2698ce6dc7be-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.773858 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.773839 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.775100 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.775067 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-envoy" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:06.777081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.777053 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-podinfo" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:39:06.777081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.777068 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-token" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:06.777260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.777146 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-kube-api-access-vm2vp" pod "78c86405-b843-4803-8b7a-2698ce6dc7be" (UID: "78c86405-b843-4803-8b7a-2698ce6dc7be"). InnerVolumeSpecName "kube-api-access-vm2vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:06.809362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.809328 5074 generic.go:356] "Generic (PLEG): container finished" podID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerID="f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208" exitCode=0 Oct 31 13:39:06.809513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.809387 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" event={"ID":"78c86405-b843-4803-8b7a-2698ce6dc7be","Type":"ContainerDied","Data":"f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208"} Oct 31 13:39:06.809513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.809401 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" Oct 31 13:39:06.809513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.809486 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs" event={"ID":"78c86405-b843-4803-8b7a-2698ce6dc7be","Type":"ContainerDied","Data":"16e050b38f0d119b9e33c24171bcc430b0c5c07d899df8915a6fbe1a7e89d31e"} Oct 31 13:39:06.809720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.809519 5074 scope.go:122] "RemoveContainer" containerID="f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208" Oct 31 13:39:06.824870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.824847 5074 scope.go:122] "RemoveContainer" containerID="df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca" Oct 31 13:39:06.838801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.838771 5074 scope.go:122] "RemoveContainer" containerID="f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208" Oct 31 13:39:06.839161 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:39:06.839135 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208\": container with ID starting with f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208 not found: ID does not exist" containerID="f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208" Oct 31 13:39:06.839218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.839170 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208"} err="failed to get container status \"f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208\": rpc error: code = NotFound desc = could not find container \"f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208\": container with ID starting with f2cc238ac26734737f4be318dcde3139c8412108fbc4f8bfa27daa8a5bc11208 not found: ID does not exist" Oct 31 13:39:06.839218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.839188 5074 scope.go:122] "RemoveContainer" containerID="df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca" Oct 31 13:39:06.839476 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:39:06.839455 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca\": container with ID starting with df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca not found: ID does not exist" containerID="df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca" Oct 31 13:39:06.839560 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.839485 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca"} err="failed to get container status \"df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca\": rpc error: code = NotFound desc = could not find container \"df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca\": container with ID starting with df91948197708d382073d343872db3d619678bf1cd1cf77e8848eab8289dfcca not found: ID does not exist" Oct 31 13:39:06.841787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.841766 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs"] Oct 31 13:39:06.854438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.854415 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e7896150-openshift-default-744c455f69-jd2fs"] Oct 31 13:39:06.875060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.875002 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.875060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.875027 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.875060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.875039 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:06.875060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:06.875050 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm2vp\" (UniqueName: \"kubernetes.io/projected/78c86405-b843-4803-8b7a-2698ce6dc7be-kube-api-access-vm2vp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:07.838648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.838599 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq"] Oct 31 13:39:07.839256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839239 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f4d7431e-9744-41b5-9c3e-25d4bdd552fa" containerName="maas-api" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839257 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d7431e-9744-41b5-9c3e-25d4bdd552fa" containerName="maas-api" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839272 5074 cpu_manager.go:414] "containerMap: removing container" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839277 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839295 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" containerName="payload-processing" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839300 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" containerName="payload-processing" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839307 5074 cpu_manager.go:414] "containerMap: removing container" podUID="07ba542c-566c-4c81-970f-969d0a6e7153" containerName="revoke-keys" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839312 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ba542c-566c-4c81-970f-969d0a6e7153" containerName="revoke-keys" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839325 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" Oct 31 13:39:07.839337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839331 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839349 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7af7844c-8037-47ed-8593-84a92852c243" containerName="payload-pre-processing" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839354 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af7844c-8037-47ed-8593-84a92852c243" containerName="payload-pre-processing" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839368 5074 cpu_manager.go:414] "containerMap: removing container" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839373 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839380 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" containerName="payload-pre-processing" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839386 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" containerName="payload-pre-processing" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839478 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af7844c-8037-47ed-8593-84a92852c243" containerName="payload-pre-processing" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839486 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839496 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839501 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="601212db-06e4-49b6-b027-5d37562ca8c2" containerName="istio-proxy" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839507 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e149ec92-34f0-4b3a-bb1a-fd851cdc8b04" containerName="payload-pre-processing" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839515 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d7431e-9744-41b5-9c3e-25d4bdd552fa" containerName="maas-api" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839523 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54bd3b0-5e0a-4b1c-95e6-53d65b44bb80" containerName="payload-processing" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839529 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ba542c-566c-4c81-970f-969d0a6e7153" containerName="revoke-keys" Oct 31 13:39:07.839667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.839535 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" Oct 31 13:39:07.844475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.844460 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.848861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.848840 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1e72f6-openshift-default-dockercfg-b6b7d\"" Oct 31 13:39:07.866277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.866236 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq"] Oct 31 13:39:07.941254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.941218 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" path="/var/lib/kubelet/pods/78c86405-b843-4803-8b7a-2698ce6dc7be/volumes" Oct 31 13:39:07.989870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.989834 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.989891 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.989995 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.990081 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.990163 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.990204 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.990249 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.990376 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:07.990612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:07.990591 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st2bn\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-kube-api-access-st2bn\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.091831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.091732 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.091831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.091786 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.091831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.091829 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.092158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.091860 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.092158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.091911 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.092158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.091945 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.092158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.092017 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.092158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.092142 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.092439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.092326 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st2bn\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-kube-api-access-st2bn\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.092900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.092868 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.093176 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.093149 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istiod-ca-cert\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.093284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.093151 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-data\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.093284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.093240 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-certs\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.093705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.093682 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-credential-socket\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.095783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.095761 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-podinfo\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.095939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.095913 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-envoy\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.106022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.105989 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st2bn\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-kube-api-access-st2bn\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.106202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.106181 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-token\") pod \"e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.159026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.158998 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:08.492586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.492556 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq"] Oct 31 13:39:08.496116 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:39:08.496071 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5a45d2_e05f_4d34_9bdd_36fb89b8d36a.slice/crio-12d954c73085790eae8cdd6b41cf3887a110a423870937cd5e6cc9360e6ca27b WatchSource:0}: Error finding container 12d954c73085790eae8cdd6b41cf3887a110a423870937cd5e6cc9360e6ca27b: Status 404 returned error can't find the container with id 12d954c73085790eae8cdd6b41cf3887a110a423870937cd5e6cc9360e6ca27b Oct 31 13:39:08.499326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.499289 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:39:08.499444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.499362 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:39:08.499444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.499392 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:39:08.829019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.828924 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" event={"ID":"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a","Type":"ContainerStarted","Data":"b4824234367a5a2f72ef7ff84df4abf9d76a0c41d0c0ae7542c489619067c9cc"} Oct 31 13:39:08.829019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.828997 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" event={"ID":"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a","Type":"ContainerStarted","Data":"12d954c73085790eae8cdd6b41cf3887a110a423870937cd5e6cc9360e6ca27b"} Oct 31 13:39:08.854765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:08.854708 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podStartSLOduration=1.85469324 podStartE2EDuration="1.85469324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:08.852567155 +0000 UTC m=+4043.030285140" watchObservedRunningTime="2026-10-31 13:39:08.85469324 +0000 UTC m=+4043.032411224" Oct 31 13:39:09.159786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:09.159695 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:09.161662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:09.161620 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 13:39:09.161781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:09.161691 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 13:39:10.175654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:10.164137 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 13:39:10.175654 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:10.164205 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 13:39:11.161658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:11.161483 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 13:39:11.161658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:11.161549 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 13:39:12.160437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:12.160400 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 13:39:12.160889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:12.160463 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 13:39:13.160328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:13.160288 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 13:39:13.160511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:13.160353 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 13:39:14.159802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:14.159761 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Oct 31 13:39:14.159998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:14.159819 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Oct 31 13:39:15.166217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.166176 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:15.166731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.166680 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:15.167478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.167454 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:39:15.887754 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.887719 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz"] Oct 31 13:39:15.888579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.888552 5074 cpu_manager.go:414] "containerMap: removing container" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" Oct 31 13:39:15.888579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.888573 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" Oct 31 13:39:15.888903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.888599 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" Oct 31 13:39:15.888903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.888605 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa965c2-ad6a-4339-879f-41318554f6ce" containerName="istio-proxy" Oct 31 13:39:15.888903 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.888831 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c86405-b843-4803-8b7a-2698ce6dc7be" containerName="istio-proxy" Oct 31 13:39:15.892746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.892728 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:15.904734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.904710 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"rate-test-model-5452fc-kserve-self-signed-certs\"" Oct 31 13:39:15.905077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.905057 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"default-dockercfg-hjcmz\"" Oct 31 13:39:15.905368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.905353 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"kube-root-ca.crt\"" Oct 31 13:39:15.905507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.905496 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5452fc\"/\"openshift-service-ca.crt\"" Oct 31 13:39:15.917253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.917226 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz"] Oct 31 13:39:15.937520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.937485 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-home\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:15.937931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.937904 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tls-certs\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:15.938103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.938079 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9t4\" (UniqueName: \"kubernetes.io/projected/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-kube-api-access-gj9t4\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:15.938325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.938306 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-dshm\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:15.938507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.938489 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tmp-dir\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:15.938769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:15.938746 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-model-cache\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.040604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.040570 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tls-certs\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.040829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.040614 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9t4\" (UniqueName: \"kubernetes.io/projected/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-kube-api-access-gj9t4\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.040900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.040872 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-dshm\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.041153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.040967 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tmp-dir\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.041153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.041126 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-model-cache\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.041374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.041169 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-home\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.041444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.041418 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tmp-dir\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.041510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.041452 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-model-cache\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.041604 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.041577 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-home\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.043557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.043513 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-dshm\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.044175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.044152 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tls-certs\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.053845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.053811 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9t4\" (UniqueName: \"kubernetes.io/projected/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-kube-api-access-gj9t4\") pod \"rate-test-model-5452fc-kserve-7957bdc47c-6vqbz\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.230653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.227903 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:16.631771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.631674 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz"] Oct 31 13:39:16.993698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.989566 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" event={"ID":"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba","Type":"ContainerStarted","Data":"795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca"} Oct 31 13:39:16.993698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:16.989613 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" event={"ID":"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba","Type":"ContainerStarted","Data":"a28fb4a4df25c1745250d97c85cb9c0915242e5726794206a48f84e18a1bfc80"} Oct 31 13:39:17.232652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:17.231669 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:26.040301 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:39:26.040256 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:39:26.232276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:26.232233 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:26.292711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:26.291987 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:26.339663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:26.336705 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" podStartSLOduration=11.336686629 podStartE2EDuration="11.336686629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:17.087763955 +0000 UTC m=+4051.265481952" watchObservedRunningTime="2026-10-31 13:39:26.336686629 +0000 UTC m=+4060.514404614" Oct 31 13:39:27.090514 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:27.090480 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:39:27.496301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:27.496267 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:39:34.144744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:34.144706 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4"] Oct 31 13:39:34.145356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:34.144974 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" containerID="cri-o://84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f" gracePeriod=30 Oct 31 13:39:37.034413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:37.034375 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr"] Oct 31 13:39:37.038537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:37.038507 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5059f5ea-7nzhr"] Oct 31 13:39:37.942582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:37.942541 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7ae030-1a22-4ecf-b1a4-070435bff7b8" path="/var/lib/kubelet/pods/aa7ae030-1a22-4ecf-b1a4-070435bff7b8/volumes" Oct 31 13:39:39.446455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.446422 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:39:39.617700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.617658 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-certs\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.617700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.617707 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-socket\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.617984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.617753 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-kube-api-access-wzz97\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-kube-api-access-wzz97\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.617984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.617831 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-token\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-token\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.617984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.617977 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istiod-ca-cert\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.618152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.617994 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-certs" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:39.618152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618078 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-socket" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:39.618152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618083 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-data\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.618152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618144 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-credential-socket\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.618442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618258 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-envoy\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.618442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618282 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-data" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:39.618442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618301 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-podinfo\") pod \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\" (UID: \"127032f3-bc89-42ff-ae41-c1ff7e9b0edb\") " Oct 31 13:39:39.618442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618425 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istiod-ca-cert" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:39:39.618687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.618466 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-credential-socket" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:39.619131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.619073 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.619343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.619317 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.619474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.619346 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.619474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.619363 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.619474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.619380 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.620812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.620786 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-kube-api-access-wzz97" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "kube-api-access-wzz97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:39.621071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.621044 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-envoy" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:39:39.621169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.621147 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-token" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:39:39.621223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.621147 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-podinfo" pod "127032f3-bc89-42ff-ae41-c1ff7e9b0edb" (UID: "127032f3-bc89-42ff-ae41-c1ff7e9b0edb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:39:39.720791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.720752 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.720791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.720786 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.720791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.720797 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzz97\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-kube-api-access-wzz97\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:39.721037 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:39.720807 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127032f3-bc89-42ff-ae41-c1ff7e9b0edb-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:39:40.254952 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.254911 5074 generic.go:356] "Generic (PLEG): container finished" podID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerID="84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f" exitCode=0 Oct 31 13:39:40.255136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.254959 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" Oct 31 13:39:40.255136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.254990 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" event={"ID":"127032f3-bc89-42ff-ae41-c1ff7e9b0edb","Type":"ContainerDied","Data":"84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f"} Oct 31 13:39:40.255136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.255047 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4" event={"ID":"127032f3-bc89-42ff-ae41-c1ff7e9b0edb","Type":"ContainerDied","Data":"f92037417e4f82604d85f5376ebe458ea12990ffbe6616ee690887763511cfb7"} Oct 31 13:39:40.255136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.255076 5074 scope.go:122] "RemoveContainer" containerID="84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f" Oct 31 13:39:40.277133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.276982 5074 scope.go:122] "RemoveContainer" containerID="1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157" Oct 31 13:39:40.283769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.283737 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4"] Oct 31 13:39:40.289457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.289424 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ef744-openshift-default-78fd5c5bfc-vxxx4"] Oct 31 13:39:40.292810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.292755 5074 scope.go:122] "RemoveContainer" containerID="84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f" Oct 31 13:39:40.293118 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:39:40.293098 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f\": container with ID starting with 84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f not found: ID does not exist" containerID="84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f" Oct 31 13:39:40.293208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.293127 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f"} err="failed to get container status \"84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f\": rpc error: code = NotFound desc = could not find container \"84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f\": container with ID starting with 84da9e563dabd21e9f3999c3422e773fed561a9f406440d34c3457680277630f not found: ID does not exist" Oct 31 13:39:40.293208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.293147 5074 scope.go:122] "RemoveContainer" containerID="1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157" Oct 31 13:39:40.293418 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:39:40.293392 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157\": container with ID starting with 1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157 not found: ID does not exist" containerID="1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157" Oct 31 13:39:40.293483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:40.293425 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157"} err="failed to get container status \"1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157\": rpc error: code = NotFound desc = could not find container \"1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157\": container with ID starting with 1ddfaa8ab3e6e0a69211675277e357fc09e1afe379e955a6a4a82128ae730157 not found: ID does not exist" Oct 31 13:39:42.019661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:42.011796 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" path="/var/lib/kubelet/pods/127032f3-bc89-42ff-ae41-c1ff7e9b0edb/volumes" Oct 31 13:39:48.039120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:48.039074 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4"] Oct 31 13:39:48.053484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:48.053415 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff23c11-jrvt4"] Oct 31 13:39:49.939276 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:49.939240 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d401ef-4486-4a34-b614-8e3278fec2bc" path="/var/lib/kubelet/pods/87d401ef-4486-4a34-b614-8e3278fec2bc/volumes" Oct 31 13:39:55.988713 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:39:55.988669 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:39:58.062658 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:58.061756 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd"] Oct 31 13:39:58.065470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:58.065434 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f87e81-rwsxd"] Oct 31 13:39:59.939799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:39:59.939761 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5be180-554a-48d5-ba44-318e98170c30" path="/var/lib/kubelet/pods/1f5be180-554a-48d5-ba44-318e98170c30/volumes" Oct 31 13:40:25.986792 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:40:25.986746 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:40:29.853986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:29.853936 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:40:47.925492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:47.925437 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j"] Oct 31 13:40:47.926188 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:47.925735 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" containerID="cri-o://97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570" gracePeriod=30 Oct 31 13:40:48.921567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.921525 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp"] Oct 31 13:40:48.922672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.922649 5074 cpu_manager.go:414] "containerMap: removing container" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" Oct 31 13:40:48.922739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.922676 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" Oct 31 13:40:48.922739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.922724 5074 cpu_manager.go:414] "containerMap: removing container" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" Oct 31 13:40:48.922739 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.922733 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" Oct 31 13:40:48.922919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.922902 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" Oct 31 13:40:48.927108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.927077 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:48.936471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.936447 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:40:48.936663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.936447 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:40:48.936788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.936494 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:40:48.938676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.938656 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:40:48.942063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:48.942033 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp"] Oct 31 13:40:49.023762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.023703 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.023949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.023868 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5n6h\" (UniqueName: \"kubernetes.io/projected/7f046502-f686-4bc5-97cb-8b568b9c6d9d-kube-api-access-m5n6h\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.024012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.024000 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.125285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.125244 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.125467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.125388 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5n6h\" (UniqueName: \"kubernetes.io/projected/7f046502-f686-4bc5-97cb-8b568b9c6d9d-kube-api-access-m5n6h\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.125467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.125450 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.126179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.126152 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.127781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.127759 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.145446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.145405 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5n6h\" (UniqueName: \"kubernetes.io/projected/7f046502-f686-4bc5-97cb-8b568b9c6d9d-kube-api-access-m5n6h\") pod \"maas-api-revoke-sub-5f4d5878-r77vp\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.245655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.245596 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:49.631131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.631088 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp"] Oct 31 13:40:49.633048 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:40:49.633015 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f046502_f686_4bc5_97cb_8b568b9c6d9d.slice/crio-d64fb49bb79e00b0fa61300e5be8d57db850efdf1314e1f5c24266ff1bc9f11e WatchSource:0}: Error finding container d64fb49bb79e00b0fa61300e5be8d57db850efdf1314e1f5c24266ff1bc9f11e: Status 404 returned error can't find the container with id d64fb49bb79e00b0fa61300e5be8d57db850efdf1314e1f5c24266ff1bc9f11e Oct 31 13:40:49.911173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:49.911129 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" event={"ID":"7f046502-f686-4bc5-97cb-8b568b9c6d9d","Type":"ContainerStarted","Data":"d64fb49bb79e00b0fa61300e5be8d57db850efdf1314e1f5c24266ff1bc9f11e"} Oct 31 13:40:50.919544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:50.919454 5074 generic.go:356] "Generic (PLEG): container finished" podID="7f046502-f686-4bc5-97cb-8b568b9c6d9d" containerID="e1438167de0b1a5f1dad71d2ee31c4a5fc51a6377b12be63a60d812d958500db" exitCode=0 Oct 31 13:40:50.920026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:50.919542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" event={"ID":"7f046502-f686-4bc5-97cb-8b568b9c6d9d","Type":"ContainerDied","Data":"e1438167de0b1a5f1dad71d2ee31c4a5fc51a6377b12be63a60d812d958500db"} Oct 31 13:40:52.110148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.110120 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:52.273542 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.273496 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f046502-f686-4bc5-97cb-8b568b9c6d9d-kube-api-access-m5n6h\" (UniqueName: \"kubernetes.io/projected/7f046502-f686-4bc5-97cb-8b568b9c6d9d-kube-api-access-m5n6h\") pod \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " Oct 31 13:40:52.273932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.273905 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-xks-ca\") pod \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " Oct 31 13:40:52.274089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.273977 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-openshift-ca\") pod \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\" (UID: \"7f046502-f686-4bc5-97cb-8b568b9c6d9d\") " Oct 31 13:40:52.274693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.274661 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-openshift-ca" pod "7f046502-f686-4bc5-97cb-8b568b9c6d9d" (UID: "7f046502-f686-4bc5-97cb-8b568b9c6d9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:52.275912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.275888 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f046502-f686-4bc5-97cb-8b568b9c6d9d-kube-api-access-m5n6h" pod "7f046502-f686-4bc5-97cb-8b568b9c6d9d" (UID: "7f046502-f686-4bc5-97cb-8b568b9c6d9d"). InnerVolumeSpecName "kube-api-access-m5n6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:52.276048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.275977 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-xks-ca" pod "7f046502-f686-4bc5-97cb-8b568b9c6d9d" (UID: "7f046502-f686-4bc5-97cb-8b568b9c6d9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:40:52.389921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.382941 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5n6h\" (UniqueName: \"kubernetes.io/projected/7f046502-f686-4bc5-97cb-8b568b9c6d9d-kube-api-access-m5n6h\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:52.389921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.382980 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:52.389921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.382996 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f046502-f686-4bc5-97cb-8b568b9c6d9d-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:52.952683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.952187 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" Oct 31 13:40:52.952683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.952229 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp" event={"ID":"7f046502-f686-4bc5-97cb-8b568b9c6d9d","Type":"ContainerDied","Data":"d64fb49bb79e00b0fa61300e5be8d57db850efdf1314e1f5c24266ff1bc9f11e"} Oct 31 13:40:52.952683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:52.952329 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64fb49bb79e00b0fa61300e5be8d57db850efdf1314e1f5c24266ff1bc9f11e" Oct 31 13:40:53.256728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.256699 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:40:53.305789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.305752 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-envoy\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.305870 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istiod-ca-cert\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306066 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-data\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306118 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-certs\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306160 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-podinfo\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306290 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306220 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-credential-socket\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306404 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306293 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-socket\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306404 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306305 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istiod-ca-cert" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:53.306509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306404 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-data" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:53.306509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306417 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-kube-api-access-n77wh\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-kube-api-access-n77wh\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306445 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-token\") pod \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\" (UID: \"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce\") " Oct 31 13:40:53.306708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306611 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-certs" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:53.306837 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.306814 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-socket" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:53.307166 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.307138 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.307251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.307170 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.307251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.307185 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.307251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.307199 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.307430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.307410 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-credential-socket" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:53.309554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.309521 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-envoy" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:53.309554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.309539 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-podinfo" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:40:53.309810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.309690 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-kube-api-access-n77wh" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "kube-api-access-n77wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:53.309810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.309769 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-token" pod "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" (UID: "bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:53.380241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.380211 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv"] Oct 31 13:40:53.406081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.406048 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x"] Oct 31 13:40:53.409216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.409182 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.409374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.409217 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n77wh\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-kube-api-access-n77wh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.409374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.409241 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.409374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.409296 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.409374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.409310 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:53.961239 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.961202 5074 generic.go:356] "Generic (PLEG): container finished" podID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerID="97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570" exitCode=0 Oct 31 13:40:53.961466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.961410 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" event={"ID":"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce","Type":"ContainerDied","Data":"97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570"} Oct 31 13:40:53.961466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.961456 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" event={"ID":"bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce","Type":"ContainerDied","Data":"40935d1e9f28af8d09b4f14dd99b8c3949a242ce849dfe34600992ac01f9b82c"} Oct 31 13:40:53.961586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.961478 5074 scope.go:122] "RemoveContainer" containerID="97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570" Oct 31 13:40:53.961677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.961604 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j" Oct 31 13:40:53.973398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.973375 5074 scope.go:122] "RemoveContainer" containerID="97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570" Oct 31 13:40:53.973718 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:40:53.973694 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570\": container with ID starting with 97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570 not found: ID does not exist" containerID="97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570" Oct 31 13:40:53.973798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.973726 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570"} err="failed to get container status \"97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570\": rpc error: code = NotFound desc = could not find container \"97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570\": container with ID starting with 97b851cf5f50a429f2851135eac4a8322c64e26aef477fa8a677a20319e80570 not found: ID does not exist" Oct 31 13:40:53.984282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.983703 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j"] Oct 31 13:40:53.988557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:53.988531 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-377f60-openshift-default-77c6686859-k764j"] Oct 31 13:40:55.939364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:55.939333 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" path="/var/lib/kubelet/pods/bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce/volumes" Oct 31 13:40:55.984573 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:40:55.984536 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:40:56.187580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:56.187547 5074 scope.go:122] "RemoveContainer" containerID="87fd37db3c4a3d0274eff16d6d46b8e00cb937932f109fb6d7bc8f98ab9bfad9" Oct 31 13:40:56.199321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:56.199299 5074 scope.go:122] "RemoveContainer" containerID="26c15e593f6b5df00d9fd2c926709f7748acfb9ff10f0103530ea74a7d990a71" Oct 31 13:40:56.210244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:40:56.210219 5074 scope.go:122] "RemoveContainer" containerID="53e39f755a90ad9f0c990cbf7090a9c4a5ca4d00a93118299272f3a70b5c094f" Oct 31 13:41:00.113953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.113913 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7"] Oct 31 13:41:00.563033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.562999 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb"] Oct 31 13:41:00.564026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.563999 5074 cpu_manager.go:414] "containerMap: removing container" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" Oct 31 13:41:00.564172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.564159 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" Oct 31 13:41:00.564244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.564233 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7f046502-f686-4bc5-97cb-8b568b9c6d9d" containerName="revoke-keys" Oct 31 13:41:00.564307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.564297 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f046502-f686-4bc5-97cb-8b568b9c6d9d" containerName="revoke-keys" Oct 31 13:41:00.564978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.564531 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="127032f3-bc89-42ff-ae41-c1ff7e9b0edb" containerName="istio-proxy" Oct 31 13:41:00.565172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.565154 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb6d3b4-940c-4d1d-afe5-645a4ab2a6ce" containerName="istio-proxy" Oct 31 13:41:00.565265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.565255 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f046502-f686-4bc5-97cb-8b568b9c6d9d" containerName="revoke-keys" Oct 31 13:41:00.571661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.571620 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb"] Oct 31 13:41:00.571942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.571878 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.578612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.578588 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:41:00.578882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.578864 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:41:00.579109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.579092 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:41:00.580721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.580700 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:41:00.624434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.624407 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.624607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.624576 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96xt\" (UniqueName: \"kubernetes.io/projected/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-kube-api-access-s96xt\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.624716 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.624609 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.725976 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.725929 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96xt\" (UniqueName: \"kubernetes.io/projected/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-kube-api-access-s96xt\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.726189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.725989 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.726257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.726229 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.726731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.726703 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.728984 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.728934 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.739209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.739179 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96xt\" (UniqueName: \"kubernetes.io/projected/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-kube-api-access-s96xt\") pod \"maas-api-revoke-sub-9d345050-m78cb\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:00.896376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:00.896277 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:01.179243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:01.179203 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb"] Oct 31 13:41:01.179594 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:41:01.179262 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeffeb1_bb62_41de_b0c6_805aa3ed1f4f.slice/crio-cb583f951f965ef35d427e7c8d0fe10b4bf5c846f3b38513af6c09f401df9af9 WatchSource:0}: Error finding container cb583f951f965ef35d427e7c8d0fe10b4bf5c846f3b38513af6c09f401df9af9: Status 404 returned error can't find the container with id cb583f951f965ef35d427e7c8d0fe10b4bf5c846f3b38513af6c09f401df9af9 Oct 31 13:41:02.053624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:02.053585 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" event={"ID":"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f","Type":"ContainerStarted","Data":"cfe086beb0751848fb0d7ca2cbac82f06489dfb1b6654eeac19547a908cecec4"} Oct 31 13:41:02.053804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:02.053676 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" event={"ID":"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f","Type":"ContainerStarted","Data":"cb583f951f965ef35d427e7c8d0fe10b4bf5c846f3b38513af6c09f401df9af9"} Oct 31 13:41:02.077891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:02.077836 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" podStartSLOduration=2.077820011 podStartE2EDuration="2.077820011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:41:02.076979942 +0000 UTC m=+4156.254697926" watchObservedRunningTime="2026-10-31 13:41:02.077820011 +0000 UTC m=+4156.255537996" Oct 31 13:41:03.062524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:03.062490 5074 generic.go:356] "Generic (PLEG): container finished" podID="7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" containerID="cfe086beb0751848fb0d7ca2cbac82f06489dfb1b6654eeac19547a908cecec4" exitCode=0 Oct 31 13:41:03.062980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:03.062540 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" event={"ID":"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f","Type":"ContainerDied","Data":"cfe086beb0751848fb0d7ca2cbac82f06489dfb1b6654eeac19547a908cecec4"} Oct 31 13:41:04.260254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.260228 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:04.386034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.385928 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-xks-ca\") pod \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " Oct 31 13:41:04.386034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.385989 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-openshift-ca\") pod \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " Oct 31 13:41:04.386284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.386108 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-kube-api-access-s96xt\" (UniqueName: \"kubernetes.io/projected/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-kube-api-access-s96xt\") pod \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\" (UID: \"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f\") " Oct 31 13:41:04.386490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.386461 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-openshift-ca" pod "7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" (UID: "7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:04.386833 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.386809 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:04.388269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.388231 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-xks-ca" pod "7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" (UID: "7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:04.388398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.388350 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-kube-api-access-s96xt" pod "7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" (UID: "7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f"). InnerVolumeSpecName "kube-api-access-s96xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:04.488408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.488371 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:04.488408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:04.488411 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96xt\" (UniqueName: \"kubernetes.io/projected/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f-kube-api-access-s96xt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:05.090702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:05.090567 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" Oct 31 13:41:05.090702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:05.090599 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb" event={"ID":"7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f","Type":"ContainerDied","Data":"cb583f951f965ef35d427e7c8d0fe10b4bf5c846f3b38513af6c09f401df9af9"} Oct 31 13:41:05.090702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:05.090673 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb583f951f965ef35d427e7c8d0fe10b4bf5c846f3b38513af6c09f401df9af9" Oct 31 13:41:05.115174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:05.115131 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" podUID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" containerName="payload-pre-processing" containerID="cri-o://ebd07b745bad1da57fa73d41873bb8cb1e6749bff836f0add78431c3ef7e2b85" gracePeriod=25 Oct 31 13:41:08.134137 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.134097 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" podUID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:08.402981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.402877 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" containerID="cri-o://44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9" gracePeriod=45 Oct 31 13:41:08.430428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.430389 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" containerID="cri-o://97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187" gracePeriod=45 Oct 31 13:41:08.739263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.735449 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 13:41:08.793321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.792936 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 13:41:08.853861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.853821 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cbee9f-cd84-4c3b-b818-0ca742df8a11-kube-api-access-vpjhh\" (UniqueName: \"kubernetes.io/projected/93cbee9f-cd84-4c3b-b818-0ca742df8a11-kube-api-access-vpjhh\") pod \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " Oct 31 13:41:08.854381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.854352 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tmp-dir\") pod \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " Oct 31 13:41:08.854444 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.854422 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-dshm\") pod \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " Oct 31 13:41:08.854501 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.854485 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-home\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-home\") pod \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " Oct 31 13:41:08.854586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.854566 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-model-cache\") pod \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " Oct 31 13:41:08.854766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.854622 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tls-certs\" (UniqueName: \"kubernetes.io/secret/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tls-certs\") pod \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\" (UID: \"93cbee9f-cd84-4c3b-b818-0ca742df8a11\") " Oct 31 13:41:08.855112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.854621 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tmp-dir" pod "93cbee9f-cd84-4c3b-b818-0ca742df8a11" (UID: "93cbee9f-cd84-4c3b-b818-0ca742df8a11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.855112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.854768 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-home" pod "93cbee9f-cd84-4c3b-b818-0ca742df8a11" (UID: "93cbee9f-cd84-4c3b-b818-0ca742df8a11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.855112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.855108 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-model-cache" pod "93cbee9f-cd84-4c3b-b818-0ca742df8a11" (UID: "93cbee9f-cd84-4c3b-b818-0ca742df8a11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.855436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.855393 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.855436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.855409 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.855436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.855417 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.856752 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.856727 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cbee9f-cd84-4c3b-b818-0ca742df8a11-kube-api-access-vpjhh" pod "93cbee9f-cd84-4c3b-b818-0ca742df8a11" (UID: "93cbee9f-cd84-4c3b-b818-0ca742df8a11"). InnerVolumeSpecName "kube-api-access-vpjhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:08.857259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.857243 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-dshm" pod "93cbee9f-cd84-4c3b-b818-0ca742df8a11" (UID: "93cbee9f-cd84-4c3b-b818-0ca742df8a11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.857543 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.857527 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tls-certs" pod "93cbee9f-cd84-4c3b-b818-0ca742df8a11" (UID: "93cbee9f-cd84-4c3b-b818-0ca742df8a11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:08.972625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.963008 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be21f7d2-5f5f-4592-afdd-591be409c854-tls-certs\" (UniqueName: \"kubernetes.io/secret/be21f7d2-5f5f-4592-afdd-591be409c854-tls-certs\") pod \"be21f7d2-5f5f-4592-afdd-591be409c854\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " Oct 31 13:41:08.972625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.963089 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-model-cache\") pod \"be21f7d2-5f5f-4592-afdd-591be409c854\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " Oct 31 13:41:08.972625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.963120 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-home\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-home\") pod \"be21f7d2-5f5f-4592-afdd-591be409c854\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " Oct 31 13:41:08.972625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.963243 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-dshm\") pod \"be21f7d2-5f5f-4592-afdd-591be409c854\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " Oct 31 13:41:08.972625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.963332 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be21f7d2-5f5f-4592-afdd-591be409c854-kube-api-access-6qzhh\" (UniqueName: \"kubernetes.io/projected/be21f7d2-5f5f-4592-afdd-591be409c854-kube-api-access-6qzhh\") pod \"be21f7d2-5f5f-4592-afdd-591be409c854\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " Oct 31 13:41:08.972625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.963384 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-tmp-dir\") pod \"be21f7d2-5f5f-4592-afdd-591be409c854\" (UID: \"be21f7d2-5f5f-4592-afdd-591be409c854\") " Oct 31 13:41:08.972625 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.971747 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-home" pod "be21f7d2-5f5f-4592-afdd-591be409c854" (UID: "be21f7d2-5f5f-4592-afdd-591be409c854"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.977825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.977251 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be21f7d2-5f5f-4592-afdd-591be409c854-tls-certs" pod "be21f7d2-5f5f-4592-afdd-591be409c854" (UID: "be21f7d2-5f5f-4592-afdd-591be409c854"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:08.977825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.977520 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-model-cache" pod "be21f7d2-5f5f-4592-afdd-591be409c854" (UID: "be21f7d2-5f5f-4592-afdd-591be409c854"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.985277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.984927 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-tmp-dir" pod "be21f7d2-5f5f-4592-afdd-591be409c854" (UID: "be21f7d2-5f5f-4592-afdd-591be409c854"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.988669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.986299 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjhh\" (UniqueName: \"kubernetes.io/projected/93cbee9f-cd84-4c3b-b818-0ca742df8a11-kube-api-access-vpjhh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.988669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.986336 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be21f7d2-5f5f-4592-afdd-591be409c854-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.988669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.986350 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.988669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.986364 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.988669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.986377 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cbee9f-cd84-4c3b-b818-0ca742df8a11-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.988669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.986389 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.988669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.986402 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cbee9f-cd84-4c3b-b818-0ca742df8a11-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:08.993614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.993443 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-dshm" pod "be21f7d2-5f5f-4592-afdd-591be409c854" (UID: "be21f7d2-5f5f-4592-afdd-591be409c854"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:08.993614 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:08.993555 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be21f7d2-5f5f-4592-afdd-591be409c854-kube-api-access-6qzhh" pod "be21f7d2-5f5f-4592-afdd-591be409c854" (UID: "be21f7d2-5f5f-4592-afdd-591be409c854"). InnerVolumeSpecName "kube-api-access-6qzhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:09.088198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.088160 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be21f7d2-5f5f-4592-afdd-591be409c854-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:09.088198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.088199 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzhh\" (UniqueName: \"kubernetes.io/projected/be21f7d2-5f5f-4592-afdd-591be409c854-kube-api-access-6qzhh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:09.136804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.136768 5074 generic.go:356] "Generic (PLEG): container finished" podID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerID="44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9" exitCode=0 Oct 31 13:41:09.137226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.136804 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" Oct 31 13:41:09.137226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.136856 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" event={"ID":"93cbee9f-cd84-4c3b-b818-0ca742df8a11","Type":"ContainerDied","Data":"44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9"} Oct 31 13:41:09.137226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.136898 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv" event={"ID":"93cbee9f-cd84-4c3b-b818-0ca742df8a11","Type":"ContainerDied","Data":"d4b42fe9bfcbcb97bd72ce205fea0dfdc5058e4019adab419108cc8fc632a58c"} Oct 31 13:41:09.137226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.136917 5074 scope.go:122] "RemoveContainer" containerID="44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9" Oct 31 13:41:09.138297 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.138278 5074 generic.go:356] "Generic (PLEG): container finished" podID="be21f7d2-5f5f-4592-afdd-591be409c854" containerID="97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187" exitCode=0 Oct 31 13:41:09.138417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.138308 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" event={"ID":"be21f7d2-5f5f-4592-afdd-591be409c854","Type":"ContainerDied","Data":"97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187"} Oct 31 13:41:09.138417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.138324 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" Oct 31 13:41:09.138417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.138348 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x" event={"ID":"be21f7d2-5f5f-4592-afdd-591be409c854","Type":"ContainerDied","Data":"3c7817c1995b4a935c3c9931b89bf859b0553b147daa117e88789f26a1e23d4f"} Oct 31 13:41:09.152973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.152956 5074 scope.go:122] "RemoveContainer" containerID="44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9" Oct 31 13:41:09.153250 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:09.153232 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9\": container with ID starting with 44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9 not found: ID does not exist" containerID="44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9" Oct 31 13:41:09.153312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.153257 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9"} err="failed to get container status \"44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9\": rpc error: code = NotFound desc = could not find container \"44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9\": container with ID starting with 44f83f47f8c3b05088a1cfec7a448c3a5dbe7012070c396f9a5d50476861afb9 not found: ID does not exist" Oct 31 13:41:09.153312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.153273 5074 scope.go:122] "RemoveContainer" containerID="97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187" Oct 31 13:41:09.162156 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.162141 5074 scope.go:122] "RemoveContainer" containerID="97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187" Oct 31 13:41:09.162406 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:09.162388 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187\": container with ID starting with 97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187 not found: ID does not exist" containerID="97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187" Oct 31 13:41:09.162467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.162411 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187"} err="failed to get container status \"97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187\": rpc error: code = NotFound desc = could not find container \"97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187\": container with ID starting with 97b5b89e31a5edf7020797f1ee89494a159af91f79514fde19a9e13c3542c187 not found: ID does not exist" Oct 31 13:41:09.167999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.167972 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv"] Oct 31 13:41:09.172201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.172178 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-107955/distinct-w0-107955-kserve-64bf5b9d-ck2qv"] Oct 31 13:41:09.183824 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.183806 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x"] Oct 31 13:41:09.191149 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.191126 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-107955/simulator-w0-107955-kserve-674f8fcd98-25x5x"] Oct 31 13:41:09.943016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.942984 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" path="/var/lib/kubelet/pods/93cbee9f-cd84-4c3b-b818-0ca742df8a11/volumes" Oct 31 13:41:09.943483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.943468 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" path="/var/lib/kubelet/pods/be21f7d2-5f5f-4592-afdd-591be409c854/volumes" Oct 31 13:41:09.998920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:09.998884 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:41:18.129197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:18.129154 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" podUID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:25.328491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.328208 5074 generic.go:356] "Generic (PLEG): container finished" podID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" containerID="ebd07b745bad1da57fa73d41873bb8cb1e6749bff836f0add78431c3ef7e2b85" exitCode=0 Oct 31 13:41:25.328491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.328417 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" event={"ID":"d1cc28c0-cb1c-42f0-b194-b370e52f8490","Type":"ContainerDied","Data":"ebd07b745bad1da57fa73d41873bb8cb1e6749bff836f0add78431c3ef7e2b85"} Oct 31 13:41:25.402623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.402593 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 13:41:25.521926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.521886 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cc28c0-cb1c-42f0-b194-b370e52f8490-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1cc28c0-cb1c-42f0-b194-b370e52f8490-plugins-config-volume\") pod \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " Oct 31 13:41:25.522116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.522041 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cc28c0-cb1c-42f0-b194-b370e52f8490-kube-api-access-pvlrf\" (UniqueName: \"kubernetes.io/projected/d1cc28c0-cb1c-42f0-b194-b370e52f8490-kube-api-access-pvlrf\") pod \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\" (UID: \"d1cc28c0-cb1c-42f0-b194-b370e52f8490\") " Oct 31 13:41:25.522316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.522291 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cc28c0-cb1c-42f0-b194-b370e52f8490-plugins-config-volume" pod "d1cc28c0-cb1c-42f0-b194-b370e52f8490" (UID: "d1cc28c0-cb1c-42f0-b194-b370e52f8490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:25.522565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.522549 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1cc28c0-cb1c-42f0-b194-b370e52f8490-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:25.524258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.524237 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cc28c0-cb1c-42f0-b194-b370e52f8490-kube-api-access-pvlrf" pod "d1cc28c0-cb1c-42f0-b194-b370e52f8490" (UID: "d1cc28c0-cb1c-42f0-b194-b370e52f8490"). InnerVolumeSpecName "kube-api-access-pvlrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:25.624253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:25.624213 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlrf\" (UniqueName: \"kubernetes.io/projected/d1cc28c0-cb1c-42f0-b194-b370e52f8490-kube-api-access-pvlrf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:25.983782 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:25.983748 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:41:26.337159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:26.337125 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" Oct 31 13:41:26.337707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:26.337151 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7" event={"ID":"d1cc28c0-cb1c-42f0-b194-b370e52f8490","Type":"ContainerDied","Data":"56d301f3287bef92dea6394e7cdde01bdd8c509d2ecd71492f2c4377c4803480"} Oct 31 13:41:26.337707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:26.337226 5074 scope.go:122] "RemoveContainer" containerID="ebd07b745bad1da57fa73d41873bb8cb1e6749bff836f0add78431c3ef7e2b85" Oct 31 13:41:26.364292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:26.364257 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7"] Oct 31 13:41:26.372378 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:26.372353 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-107955-b67b679f-2nlj7"] Oct 31 13:41:27.941151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:27.941108 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" path="/var/lib/kubelet/pods/d1cc28c0-cb1c-42f0-b194-b370e52f8490/volumes" Oct 31 13:41:31.138655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:31.134598 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp"] Oct 31 13:41:31.161477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:31.161432 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k"] Oct 31 13:41:32.025710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:32.025672 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74"] Oct 31 13:41:32.030314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:32.030282 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f00da7-kpm74"] Oct 31 13:41:33.939612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:33.939566 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef5ff65-f1c8-4cdd-8302-05fb77e02f29" path="/var/lib/kubelet/pods/9ef5ff65-f1c8-4cdd-8302-05fb77e02f29/volumes" Oct 31 13:41:35.151162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:35.151115 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj"] Oct 31 13:41:35.151686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:35.151322 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" containerID="cri-o://e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579" gracePeriod=30 Oct 31 13:41:36.135732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:36.135690 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" podUID="cc8a91e2-db37-4692-b1c2-de0c1909e825" containerName="payload-processing" containerID="cri-o://5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238" gracePeriod=25 Oct 31 13:41:36.162041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:36.161991 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" podUID="faa58097-a14b-4f50-81fe-b2d3265f4d29" containerName="payload-pre-processing" containerID="cri-o://3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef" gracePeriod=25 Oct 31 13:41:38.043466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:38.043430 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl"] Oct 31 13:41:38.048117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:38.048088 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4732e9d5-zbckl"] Oct 31 13:41:39.939423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:39.939387 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6329f68a-deb4-4bb7-bd8d-d7ddc805a479" path="/var/lib/kubelet/pods/6329f68a-deb4-4bb7-bd8d-d7ddc805a479/volumes" Oct 31 13:41:40.433680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.433655 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 13:41:40.522584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.522110 5074 generic.go:356] "Generic (PLEG): container finished" podID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerID="e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579" exitCode=0 Oct 31 13:41:40.522584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.522265 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" event={"ID":"c39ca4bf-f2b7-4615-93d1-ace6e650716d","Type":"ContainerDied","Data":"e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579"} Oct 31 13:41:40.522584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.522320 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" event={"ID":"c39ca4bf-f2b7-4615-93d1-ace6e650716d","Type":"ContainerDied","Data":"c0dba4a64e0f8e3b96a95f96360a6d572298297b9ea4221325cd4cc56af14fb4"} Oct 31 13:41:40.522584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.522360 5074 scope.go:122] "RemoveContainer" containerID="e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579" Oct 31 13:41:40.523030 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.522749 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj" Oct 31 13:41:40.547509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.547479 5074 scope.go:122] "RemoveContainer" containerID="af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b" Oct 31 13:41:40.561252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.561220 5074 scope.go:122] "RemoveContainer" containerID="e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579" Oct 31 13:41:40.561690 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:40.561627 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579\": container with ID starting with e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579 not found: ID does not exist" containerID="e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579" Oct 31 13:41:40.561789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.561701 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579"} err="failed to get container status \"e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579\": rpc error: code = NotFound desc = could not find container \"e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579\": container with ID starting with e55e87e0f1b1b474b18b14d73da17a65c2c66cb99950a3b0d7782da3c84c9579 not found: ID does not exist" Oct 31 13:41:40.561789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.561725 5074 scope.go:122] "RemoveContainer" containerID="af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b" Oct 31 13:41:40.562057 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:40.562035 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b\": container with ID starting with af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b not found: ID does not exist" containerID="af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b" Oct 31 13:41:40.562128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.562069 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b"} err="failed to get container status \"af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b\": rpc error: code = NotFound desc = could not find container \"af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b\": container with ID starting with af12e1fa2456be8eca27d97ef3ba498d59b41071c574db6bef9c795949fa739b not found: ID does not exist" Oct 31 13:41:40.605431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605396 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-token\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-token\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.605597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605467 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-certs\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.605597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605562 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-data\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.605773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605623 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-credential-socket\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.605844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605797 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-socket\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.605901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605853 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-certs" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:40.605969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605896 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-data" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:40.605969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.605902 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-credential-socket" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:40.606082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606043 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-socket" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:40.606140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606081 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-envoy\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.606140 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606125 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istiod-ca-cert\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.606250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606163 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-kube-api-access-sln2l\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-kube-api-access-sln2l\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.606250 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606194 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-podinfo\") pod \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\" (UID: \"c39ca4bf-f2b7-4615-93d1-ace6e650716d\") " Oct 31 13:41:40.606555 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606527 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istiod-ca-cert" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:40.606972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606948 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.606972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606972 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.606972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.606986 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.607273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.607002 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.607273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.607015 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.608971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.608941 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-token" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:40.608971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.608954 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-envoy" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:40.609091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.608954 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-kube-api-access-sln2l" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "kube-api-access-sln2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:40.609472 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.609452 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-podinfo" pod "c39ca4bf-f2b7-4615-93d1-ace6e650716d" (UID: "c39ca4bf-f2b7-4615-93d1-ace6e650716d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:41:40.708547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.708512 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.708547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.708544 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sln2l\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-kube-api-access-sln2l\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.708547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.708556 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.708855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.708565 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ca4bf-f2b7-4615-93d1-ace6e650716d-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:40.849112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.849083 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj"] Oct 31 13:41:40.857380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:40.857353 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-107955-openshift-default-6c56bcd96c-gz4hj"] Oct 31 13:41:41.959746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:41.959696 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" path="/var/lib/kubelet/pods/c39ca4bf-f2b7-4615-93d1-ace6e650716d/volumes" Oct 31 13:41:45.821055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:45.821015 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" podUID="cc8a91e2-db37-4692-b1c2-de0c1909e825" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:45.821456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:45.821072 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" podUID="faa58097-a14b-4f50-81fe-b2d3265f4d29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:47.190384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:47.190352 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:41:47.206796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:47.206768 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:41:49.034064 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:49.034028 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n"] Oct 31 13:41:49.037336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:49.037307 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc102b33-4x59n"] Oct 31 13:41:49.941887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:49.941851 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25263f3e-949f-459c-81b1-494cd89a19c1" path="/var/lib/kubelet/pods/25263f3e-949f-459c-81b1-494cd89a19c1/volumes" Oct 31 13:41:51.645880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.645843 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw"] Oct 31 13:41:51.646475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646459 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" containerName="payload-pre-processing" Oct 31 13:41:51.646527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646476 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" containerName="payload-pre-processing" Oct 31 13:41:51.646527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646490 5074 cpu_manager.go:414] "containerMap: removing container" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" Oct 31 13:41:51.646527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646496 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" Oct 31 13:41:51.646527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646504 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" containerName="revoke-keys" Oct 31 13:41:51.646527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646509 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" containerName="revoke-keys" Oct 31 13:41:51.646527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646526 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646530 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646553 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646557 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646575 5074 cpu_manager.go:414] "containerMap: removing container" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646580 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646682 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646693 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="be21f7d2-5f5f-4592-afdd-591be409c854" containerName="main" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646702 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cc28c0-cb1c-42f0-b194-b370e52f8490" containerName="payload-pre-processing" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646709 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cbee9f-cd84-4c3b-b818-0ca742df8a11" containerName="main" Oct 31 13:41:51.646736 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.646714 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" containerName="revoke-keys" Oct 31 13:41:51.651591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.651572 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.657172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.657149 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:41:51.657346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.657304 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:41:51.657346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.657176 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:41:51.658665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.658646 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:41:51.659512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.659487 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw"] Oct 31 13:41:51.755804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.755764 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.756028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.755994 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq2r\" (UniqueName: \"kubernetes.io/projected/e62dc5df-b667-4645-b4ee-a9f37dc8c901-kube-api-access-9rq2r\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.756284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.756095 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.857573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.857537 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.857790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.857614 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq2r\" (UniqueName: \"kubernetes.io/projected/e62dc5df-b667-4645-b4ee-a9f37dc8c901-kube-api-access-9rq2r\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.857790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.857647 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.858418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.858395 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.860044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.860021 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.870066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.870041 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq2r\" (UniqueName: \"kubernetes.io/projected/e62dc5df-b667-4645-b4ee-a9f37dc8c901-kube-api-access-9rq2r\") pod \"maas-api-revoke-sub-2e26e013-z9bmw\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:51.967762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:51.967737 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:52.255949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:52.255923 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw"] Oct 31 13:41:52.257309 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:41:52.257268 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62dc5df_b667_4645_b4ee_a9f37dc8c901.slice/crio-c03c389f2cf1a39be57c1d5f5428dc33b75884ec195b4a74ccac8d1bfbd04d30 WatchSource:0}: Error finding container c03c389f2cf1a39be57c1d5f5428dc33b75884ec195b4a74ccac8d1bfbd04d30: Status 404 returned error can't find the container with id c03c389f2cf1a39be57c1d5f5428dc33b75884ec195b4a74ccac8d1bfbd04d30 Oct 31 13:41:52.617516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:52.617467 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" event={"ID":"e62dc5df-b667-4645-b4ee-a9f37dc8c901","Type":"ContainerStarted","Data":"71021dac81160652d6df95a4ea5f1ab42f52c462282c6642192a1320d9383f35"} Oct 31 13:41:52.617743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:52.617538 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" event={"ID":"e62dc5df-b667-4645-b4ee-a9f37dc8c901","Type":"ContainerStarted","Data":"c03c389f2cf1a39be57c1d5f5428dc33b75884ec195b4a74ccac8d1bfbd04d30"} Oct 31 13:41:52.658422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:52.658370 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" podStartSLOduration=1.6583573679999999 podStartE2EDuration="1.658357368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:41:52.654707199 +0000 UTC m=+4206.832425182" watchObservedRunningTime="2026-10-31 13:41:52.658357368 +0000 UTC m=+4206.836075352" Oct 31 13:41:53.626890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:53.626736 5074 generic.go:356] "Generic (PLEG): container finished" podID="e62dc5df-b667-4645-b4ee-a9f37dc8c901" containerID="71021dac81160652d6df95a4ea5f1ab42f52c462282c6642192a1320d9383f35" exitCode=0 Oct 31 13:41:53.627037 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:53.626924 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" event={"ID":"e62dc5df-b667-4645-b4ee-a9f37dc8c901","Type":"ContainerDied","Data":"71021dac81160652d6df95a4ea5f1ab42f52c462282c6642192a1320d9383f35"} Oct 31 13:41:54.816443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.816413 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:54.905087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.904985 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-openshift-ca\") pod \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " Oct 31 13:41:54.905891 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.905722 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-openshift-ca" pod "e62dc5df-b667-4645-b4ee-a9f37dc8c901" (UID: "e62dc5df-b667-4645-b4ee-a9f37dc8c901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:54.906208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.906170 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-xks-ca\") pod \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " Oct 31 13:41:54.907141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.907119 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62dc5df-b667-4645-b4ee-a9f37dc8c901-kube-api-access-9rq2r\" (UniqueName: \"kubernetes.io/projected/e62dc5df-b667-4645-b4ee-a9f37dc8c901-kube-api-access-9rq2r\") pod \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\" (UID: \"e62dc5df-b667-4645-b4ee-a9f37dc8c901\") " Oct 31 13:41:54.908321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.908301 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:54.918854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.918807 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62dc5df-b667-4645-b4ee-a9f37dc8c901-kube-api-access-9rq2r" pod "e62dc5df-b667-4645-b4ee-a9f37dc8c901" (UID: "e62dc5df-b667-4645-b4ee-a9f37dc8c901"). InnerVolumeSpecName "kube-api-access-9rq2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:54.919092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:54.919055 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-xks-ca" pod "e62dc5df-b667-4645-b4ee-a9f37dc8c901" (UID: "e62dc5df-b667-4645-b4ee-a9f37dc8c901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:41:55.010601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:55.010568 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62dc5df-b667-4645-b4ee-a9f37dc8c901-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:55.010601 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:55.010603 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rq2r\" (UniqueName: \"kubernetes.io/projected/e62dc5df-b667-4645-b4ee-a9f37dc8c901-kube-api-access-9rq2r\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:55.670025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:55.669989 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" Oct 31 13:41:55.670025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:55.670004 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw" event={"ID":"e62dc5df-b667-4645-b4ee-a9f37dc8c901","Type":"ContainerDied","Data":"c03c389f2cf1a39be57c1d5f5428dc33b75884ec195b4a74ccac8d1bfbd04d30"} Oct 31 13:41:55.670252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:55.670085 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c03c389f2cf1a39be57c1d5f5428dc33b75884ec195b4a74ccac8d1bfbd04d30" Oct 31 13:41:55.820559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:55.820525 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" podUID="cc8a91e2-db37-4692-b1c2-de0c1909e825" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:55.821051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:55.820555 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" podUID="faa58097-a14b-4f50-81fe-b2d3265f4d29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:41:55.985587 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:55.985542 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:41:56.507759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.502180 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:41:56.537945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.537281 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8a91e2-db37-4692-b1c2-de0c1909e825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a91e2-db37-4692-b1c2-de0c1909e825-plugins-config-volume\") pod \"cc8a91e2-db37-4692-b1c2-de0c1909e825\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " Oct 31 13:41:56.537945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.537604 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8a91e2-db37-4692-b1c2-de0c1909e825-kube-api-access-j82q6\" (UniqueName: \"kubernetes.io/projected/cc8a91e2-db37-4692-b1c2-de0c1909e825-kube-api-access-j82q6\") pod \"cc8a91e2-db37-4692-b1c2-de0c1909e825\" (UID: \"cc8a91e2-db37-4692-b1c2-de0c1909e825\") " Oct 31 13:41:56.539109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.539077 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8a91e2-db37-4692-b1c2-de0c1909e825-plugins-config-volume" pod "cc8a91e2-db37-4692-b1c2-de0c1909e825" (UID: "cc8a91e2-db37-4692-b1c2-de0c1909e825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:56.541356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.541325 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8a91e2-db37-4692-b1c2-de0c1909e825-kube-api-access-j82q6" pod "cc8a91e2-db37-4692-b1c2-de0c1909e825" (UID: "cc8a91e2-db37-4692-b1c2-de0c1909e825"). InnerVolumeSpecName "kube-api-access-j82q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:56.564717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.564695 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:41:56.640196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.640161 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa58097-a14b-4f50-81fe-b2d3265f4d29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa58097-a14b-4f50-81fe-b2d3265f4d29-plugins-config-volume\") pod \"faa58097-a14b-4f50-81fe-b2d3265f4d29\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " Oct 31 13:41:56.640401 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.640260 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa58097-a14b-4f50-81fe-b2d3265f4d29-kube-api-access-r2nl6\" (UniqueName: \"kubernetes.io/projected/faa58097-a14b-4f50-81fe-b2d3265f4d29-kube-api-access-r2nl6\") pod \"faa58097-a14b-4f50-81fe-b2d3265f4d29\" (UID: \"faa58097-a14b-4f50-81fe-b2d3265f4d29\") " Oct 31 13:41:56.640592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.640567 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa58097-a14b-4f50-81fe-b2d3265f4d29-plugins-config-volume" pod "faa58097-a14b-4f50-81fe-b2d3265f4d29" (UID: "faa58097-a14b-4f50-81fe-b2d3265f4d29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:56.640827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.640804 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa58097-a14b-4f50-81fe-b2d3265f4d29-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:56.640827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.640828 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a91e2-db37-4692-b1c2-de0c1909e825-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:56.641014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.640838 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j82q6\" (UniqueName: \"kubernetes.io/projected/cc8a91e2-db37-4692-b1c2-de0c1909e825-kube-api-access-j82q6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:56.643006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.642974 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa58097-a14b-4f50-81fe-b2d3265f4d29-kube-api-access-r2nl6" pod "faa58097-a14b-4f50-81fe-b2d3265f4d29" (UID: "faa58097-a14b-4f50-81fe-b2d3265f4d29"). InnerVolumeSpecName "kube-api-access-r2nl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:56.679045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.679013 5074 generic.go:356] "Generic (PLEG): container finished" podID="faa58097-a14b-4f50-81fe-b2d3265f4d29" containerID="3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef" exitCode=0 Oct 31 13:41:56.679187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.679061 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" Oct 31 13:41:56.679187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.679095 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" event={"ID":"faa58097-a14b-4f50-81fe-b2d3265f4d29","Type":"ContainerDied","Data":"3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef"} Oct 31 13:41:56.679187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.679156 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k" event={"ID":"faa58097-a14b-4f50-81fe-b2d3265f4d29","Type":"ContainerDied","Data":"5141df6fcfe823fcfaa8e7e1b70282dbc1872e64884d6d513b5e2a46a3e7ba47"} Oct 31 13:41:56.679334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.679187 5074 scope.go:122] "RemoveContainer" containerID="3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef" Oct 31 13:41:56.680762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.680722 5074 generic.go:356] "Generic (PLEG): container finished" podID="cc8a91e2-db37-4692-b1c2-de0c1909e825" containerID="5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238" exitCode=0 Oct 31 13:41:56.680883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.680770 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" Oct 31 13:41:56.680883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.680817 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" event={"ID":"cc8a91e2-db37-4692-b1c2-de0c1909e825","Type":"ContainerDied","Data":"5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238"} Oct 31 13:41:56.680883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.680854 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp" event={"ID":"cc8a91e2-db37-4692-b1c2-de0c1909e825","Type":"ContainerDied","Data":"718d64857c2db07370406ca46b0c2ae4b6d3499ff36b3d4d155534da72f26ad2"} Oct 31 13:41:56.693349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.693326 5074 scope.go:122] "RemoveContainer" containerID="3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef" Oct 31 13:41:56.693662 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:56.693610 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef\": container with ID starting with 3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef not found: ID does not exist" containerID="3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef" Oct 31 13:41:56.693738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.693666 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef"} err="failed to get container status \"3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef\": rpc error: code = NotFound desc = could not find container \"3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef\": container with ID starting with 3eaa8f9809928c4e169c6d7e21a239401ededd4d6efaf4bffcbd4217469fcfef not found: ID does not exist" Oct 31 13:41:56.693738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.693683 5074 scope.go:122] "RemoveContainer" containerID="5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238" Oct 31 13:41:56.705872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.705852 5074 scope.go:122] "RemoveContainer" containerID="5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238" Oct 31 13:41:56.706147 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:41:56.706127 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238\": container with ID starting with 5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238 not found: ID does not exist" containerID="5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238" Oct 31 13:41:56.706227 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.706152 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238"} err="failed to get container status \"5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238\": rpc error: code = NotFound desc = could not find container \"5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238\": container with ID starting with 5019e8db5988bcf831c9c1780bab7d857ddc726b75b269fe188b243f417c3238 not found: ID does not exist" Oct 31 13:41:56.713785 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.713763 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k"] Oct 31 13:41:56.719028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.719007 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-54a16b-5485f79767-j2l8k"] Oct 31 13:41:56.725768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.725737 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp"] Oct 31 13:41:56.734257 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.734237 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-54a16b-88ddcf457-8f2lp"] Oct 31 13:41:56.742306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:56.742284 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2nl6\" (UniqueName: \"kubernetes.io/projected/faa58097-a14b-4f50-81fe-b2d3265f4d29-kube-api-access-r2nl6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:57.939622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:57.939590 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8a91e2-db37-4692-b1c2-de0c1909e825" path="/var/lib/kubelet/pods/cc8a91e2-db37-4692-b1c2-de0c1909e825/volumes" Oct 31 13:41:57.940097 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:41:57.940079 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa58097-a14b-4f50-81fe-b2d3265f4d29" path="/var/lib/kubelet/pods/faa58097-a14b-4f50-81fe-b2d3265f4d29/volumes" Oct 31 13:42:00.972234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:00.972186 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:42:01.441714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.441607 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8"] Oct 31 13:42:01.442274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442252 5074 cpu_manager.go:414] "containerMap: removing container" podUID="e62dc5df-b667-4645-b4ee-a9f37dc8c901" containerName="revoke-keys" Oct 31 13:42:01.442274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442271 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62dc5df-b667-4645-b4ee-a9f37dc8c901" containerName="revoke-keys" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442297 5074 cpu_manager.go:414] "containerMap: removing container" podUID="faa58097-a14b-4f50-81fe-b2d3265f4d29" containerName="payload-pre-processing" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442303 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa58097-a14b-4f50-81fe-b2d3265f4d29" containerName="payload-pre-processing" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442319 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cc8a91e2-db37-4692-b1c2-de0c1909e825" containerName="payload-processing" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442324 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8a91e2-db37-4692-b1c2-de0c1909e825" containerName="payload-processing" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442422 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8a91e2-db37-4692-b1c2-de0c1909e825" containerName="payload-processing" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442434 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62dc5df-b667-4645-b4ee-a9f37dc8c901" containerName="revoke-keys" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442442 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39ca4bf-f2b7-4615-93d1-ace6e650716d" containerName="istio-proxy" Oct 31 13:42:01.442475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.442451 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa58097-a14b-4f50-81fe-b2d3265f4d29" containerName="payload-pre-processing" Oct 31 13:42:01.447435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.447408 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.458965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.458937 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8"] Oct 31 13:42:01.519769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.519732 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.519769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.519771 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.519985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.519910 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxgm\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-kube-api-access-hpxgm\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.519985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.519963 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.520088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.519988 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa12a2af-57d5-4270-bd66-2d8d202714cf-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.520088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.520036 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.520196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.520108 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.520256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.520196 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.520313 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.520295 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.525701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.525678 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-569075-openshift-default-dockercfg-lwqkn\"" Oct 31 13:42:01.622270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.622225 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.622717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.622661 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.623096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623066 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.623525 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623495 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-certs\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.623620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623536 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.623789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623760 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-data\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.623912 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623889 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxgm\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-kube-api-access-hpxgm\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.623983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623914 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.623983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623936 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa12a2af-57d5-4270-bd66-2d8d202714cf-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.624104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.623996 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.624104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.624023 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.624442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.624420 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.624529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.624500 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-credential-socket\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.624809 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.624789 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa12a2af-57d5-4270-bd66-2d8d202714cf-istiod-ca-cert\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.625979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.625952 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-podinfo\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.626460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.626443 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-envoy\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.640359 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.640337 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-token\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.640460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.640379 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxgm\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-kube-api-access-hpxgm\") pod \"e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.768253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.767897 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:01.961350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.961318 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc"] Oct 31 13:42:01.967021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.966998 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:01.972373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.972344 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:42:01.973077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.972660 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:42:01.975895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.975857 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:42:01.976043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.975917 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:42:01.979478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:01.978394 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc"] Oct 31 13:42:02.040874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.033694 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.049998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.049941 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsp7k\" (UniqueName: \"kubernetes.io/projected/43a3b4ce-2473-4d61-a213-f212d50d7a2a-kube-api-access-lsp7k\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.050222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.050074 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.131708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.126106 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:02.131708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.126196 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:02.131708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.126242 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:02.135691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.135136 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8"] Oct 31 13:42:02.153918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.153871 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.154931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.154901 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsp7k\" (UniqueName: \"kubernetes.io/projected/43a3b4ce-2473-4d61-a213-f212d50d7a2a-kube-api-access-lsp7k\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.155436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.155410 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.155900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.154724 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.161409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.161387 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.174252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.174159 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsp7k\" (UniqueName: \"kubernetes.io/projected/43a3b4ce-2473-4d61-a213-f212d50d7a2a-kube-api-access-lsp7k\") pod \"maas-api-revoke-sub-fc203ecb-b24jc\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.287086 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.287046 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:02.625302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.625210 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc"] Oct 31 13:42:02.628931 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:02.628899 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a3b4ce_2473_4d61_a213_f212d50d7a2a.slice/crio-272bf905386678d71701a1a4db16057ed12f4e62c61f3fa95e7401edcb685d9b WatchSource:0}: Error finding container 272bf905386678d71701a1a4db16057ed12f4e62c61f3fa95e7401edcb685d9b: Status 404 returned error can't find the container with id 272bf905386678d71701a1a4db16057ed12f4e62c61f3fa95e7401edcb685d9b Oct 31 13:42:02.805660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.804116 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" event={"ID":"fa12a2af-57d5-4270-bd66-2d8d202714cf","Type":"ContainerStarted","Data":"3d5d8ff86fc5aae05767dfd4564eb6c6ed29334ee0f85be1d9031636474acf6b"} Oct 31 13:42:02.805660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.804191 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" event={"ID":"fa12a2af-57d5-4270-bd66-2d8d202714cf","Type":"ContainerStarted","Data":"88773ecb039dd7c456085d79106be87dc5e7869e3abfc3c1a2363b594a5757a0"} Oct 31 13:42:02.807217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.807187 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" event={"ID":"43a3b4ce-2473-4d61-a213-f212d50d7a2a","Type":"ContainerStarted","Data":"272bf905386678d71701a1a4db16057ed12f4e62c61f3fa95e7401edcb685d9b"} Oct 31 13:42:02.832014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:02.831956 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" podStartSLOduration=1.8319416450000001 podStartE2EDuration="1.831941645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:02.82824493 +0000 UTC m=+4217.005962910" watchObservedRunningTime="2026-10-31 13:42:02.831941645 +0000 UTC m=+4217.009659668" Oct 31 13:42:03.784798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:03.783970 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:03.795707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:03.794766 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Oct 31 13:42:03.795707 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:03.794830 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Oct 31 13:42:03.815889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:03.815861 5074 generic.go:356] "Generic (PLEG): container finished" podID="43a3b4ce-2473-4d61-a213-f212d50d7a2a" containerID="a9b54e83239085db556af16033d552698134cbf6ba2ca0d59e771f390ddab6de" exitCode=0 Oct 31 13:42:03.816017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:03.815948 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" event={"ID":"43a3b4ce-2473-4d61-a213-f212d50d7a2a","Type":"ContainerDied","Data":"a9b54e83239085db556af16033d552698134cbf6ba2ca0d59e771f390ddab6de"} Oct 31 13:42:04.508150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.508113 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6"] Oct 31 13:42:04.512435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.512410 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.520689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.520613 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.520843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.520789 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt49p\" (UniqueName: \"kubernetes.io/projected/ece3289c-d09c-4ac5-8a62-003d291e2b14-kube-api-access-bt49p\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.520904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.520879 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.523469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.523353 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6"] Oct 31 13:42:04.622287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.622241 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt49p\" (UniqueName: \"kubernetes.io/projected/ece3289c-d09c-4ac5-8a62-003d291e2b14-kube-api-access-bt49p\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.622454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.622333 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.622518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.622492 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.623356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.623330 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.625042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.625015 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.634841 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.634811 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt49p\" (UniqueName: \"kubernetes.io/projected/ece3289c-d09c-4ac5-8a62-003d291e2b14-kube-api-access-bt49p\") pod \"maas-api-revoke-sub-1e47c272-xmvp6\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:04.790711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.777065 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Oct 31 13:42:04.790711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.777143 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Oct 31 13:42:04.836510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:04.836481 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:05.167932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.167907 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:05.231497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.231457 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-openshift-ca\") pod \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " Oct 31 13:42:05.231689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.231583 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a3b4ce-2473-4d61-a213-f212d50d7a2a-kube-api-access-lsp7k\" (UniqueName: \"kubernetes.io/projected/43a3b4ce-2473-4d61-a213-f212d50d7a2a-kube-api-access-lsp7k\") pod \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " Oct 31 13:42:05.231689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.231646 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-xks-ca\") pod \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\" (UID: \"43a3b4ce-2473-4d61-a213-f212d50d7a2a\") " Oct 31 13:42:05.233588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.233399 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-openshift-ca" pod "43a3b4ce-2473-4d61-a213-f212d50d7a2a" (UID: "43a3b4ce-2473-4d61-a213-f212d50d7a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:05.233588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.233554 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6"] Oct 31 13:42:05.235335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.235308 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-xks-ca" pod "43a3b4ce-2473-4d61-a213-f212d50d7a2a" (UID: "43a3b4ce-2473-4d61-a213-f212d50d7a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:05.236182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.236148 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a3b4ce-2473-4d61-a213-f212d50d7a2a-kube-api-access-lsp7k" pod "43a3b4ce-2473-4d61-a213-f212d50d7a2a" (UID: "43a3b4ce-2473-4d61-a213-f212d50d7a2a"). InnerVolumeSpecName "kube-api-access-lsp7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:05.237488 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:05.237435 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece3289c_d09c_4ac5_8a62_003d291e2b14.slice/crio-1e9e79f0a608191933b341303aed746a1a3a572e873ea7a7b8b11d0977b101c6 WatchSource:0}: Error finding container 1e9e79f0a608191933b341303aed746a1a3a572e873ea7a7b8b11d0977b101c6: Status 404 returned error can't find the container with id 1e9e79f0a608191933b341303aed746a1a3a572e873ea7a7b8b11d0977b101c6 Oct 31 13:42:05.342659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.339665 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:05.342659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.339704 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsp7k\" (UniqueName: \"kubernetes.io/projected/43a3b4ce-2473-4d61-a213-f212d50d7a2a-kube-api-access-lsp7k\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:05.342659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.339718 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a3b4ce-2473-4d61-a213-f212d50d7a2a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:05.785653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.776029 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Oct 31 13:42:05.785653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.776089 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Oct 31 13:42:05.833507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.833477 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" event={"ID":"ece3289c-d09c-4ac5-8a62-003d291e2b14","Type":"ContainerStarted","Data":"9bcc20205c241f783a8757dfac23ad150f553b4332aa3a2ff1021a76e09bb82b"} Oct 31 13:42:05.834070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.834049 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" event={"ID":"ece3289c-d09c-4ac5-8a62-003d291e2b14","Type":"ContainerStarted","Data":"1e9e79f0a608191933b341303aed746a1a3a572e873ea7a7b8b11d0977b101c6"} Oct 31 13:42:05.834997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.834971 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" event={"ID":"43a3b4ce-2473-4d61-a213-f212d50d7a2a","Type":"ContainerDied","Data":"272bf905386678d71701a1a4db16057ed12f4e62c61f3fa95e7401edcb685d9b"} Oct 31 13:42:05.835094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.835012 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="272bf905386678d71701a1a4db16057ed12f4e62c61f3fa95e7401edcb685d9b" Oct 31 13:42:05.835094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.835032 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc" Oct 31 13:42:05.856053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:05.854578 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" podStartSLOduration=1.854559707 podStartE2EDuration="1.854559707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:05.852472933 +0000 UTC m=+4220.030190918" watchObservedRunningTime="2026-10-31 13:42:05.854559707 +0000 UTC m=+4220.032277692" Oct 31 13:42:06.789744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:06.789651 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Oct 31 13:42:06.790014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:06.789982 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Oct 31 13:42:06.853695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:06.853652 5074 generic.go:356] "Generic (PLEG): container finished" podID="ece3289c-d09c-4ac5-8a62-003d291e2b14" containerID="9bcc20205c241f783a8757dfac23ad150f553b4332aa3a2ff1021a76e09bb82b" exitCode=0 Oct 31 13:42:06.854093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:06.853735 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" event={"ID":"ece3289c-d09c-4ac5-8a62-003d291e2b14","Type":"ContainerDied","Data":"9bcc20205c241f783a8757dfac23ad150f553b4332aa3a2ff1021a76e09bb82b"} Oct 31 13:42:07.778134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:07.778095 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body= Oct 31 13:42:07.778434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:07.778402 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" Oct 31 13:42:08.094876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.094843 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:08.116957 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.116914 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-xks-ca\") pod \"ece3289c-d09c-4ac5-8a62-003d291e2b14\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " Oct 31 13:42:08.117145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.117026 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece3289c-d09c-4ac5-8a62-003d291e2b14-kube-api-access-bt49p\" (UniqueName: \"kubernetes.io/projected/ece3289c-d09c-4ac5-8a62-003d291e2b14-kube-api-access-bt49p\") pod \"ece3289c-d09c-4ac5-8a62-003d291e2b14\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " Oct 31 13:42:08.117203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.117179 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-openshift-ca\") pod \"ece3289c-d09c-4ac5-8a62-003d291e2b14\" (UID: \"ece3289c-d09c-4ac5-8a62-003d291e2b14\") " Oct 31 13:42:08.117643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.117563 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-openshift-ca" pod "ece3289c-d09c-4ac5-8a62-003d291e2b14" (UID: "ece3289c-d09c-4ac5-8a62-003d291e2b14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:08.119836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.119806 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-xks-ca" pod "ece3289c-d09c-4ac5-8a62-003d291e2b14" (UID: "ece3289c-d09c-4ac5-8a62-003d291e2b14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:08.119951 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.119872 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece3289c-d09c-4ac5-8a62-003d291e2b14-kube-api-access-bt49p" pod "ece3289c-d09c-4ac5-8a62-003d291e2b14" (UID: "ece3289c-d09c-4ac5-8a62-003d291e2b14"). InnerVolumeSpecName "kube-api-access-bt49p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:08.219120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.219080 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:08.219120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.219116 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece3289c-d09c-4ac5-8a62-003d291e2b14-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:08.219341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.219129 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt49p\" (UniqueName: \"kubernetes.io/projected/ece3289c-d09c-4ac5-8a62-003d291e2b14-kube-api-access-bt49p\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:08.774120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.774085 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:08.774562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.774535 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:08.775299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.775276 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:42:08.873893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.873627 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" Oct 31 13:42:08.873893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.873667 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6" event={"ID":"ece3289c-d09c-4ac5-8a62-003d291e2b14","Type":"ContainerDied","Data":"1e9e79f0a608191933b341303aed746a1a3a572e873ea7a7b8b11d0977b101c6"} Oct 31 13:42:08.873893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:08.873730 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e9e79f0a608191933b341303aed746a1a3a572e873ea7a7b8b11d0977b101c6" Oct 31 13:42:12.818415 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.818379 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26"] Oct 31 13:42:12.819026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.819009 5074 cpu_manager.go:414] "containerMap: removing container" podUID="43a3b4ce-2473-4d61-a213-f212d50d7a2a" containerName="revoke-keys" Oct 31 13:42:12.819079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.819027 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a3b4ce-2473-4d61-a213-f212d50d7a2a" containerName="revoke-keys" Oct 31 13:42:12.819079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.819048 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ece3289c-d09c-4ac5-8a62-003d291e2b14" containerName="revoke-keys" Oct 31 13:42:12.819079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.819053 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece3289c-d09c-4ac5-8a62-003d291e2b14" containerName="revoke-keys" Oct 31 13:42:12.819177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.819152 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a3b4ce-2473-4d61-a213-f212d50d7a2a" containerName="revoke-keys" Oct 31 13:42:12.819177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.819162 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece3289c-d09c-4ac5-8a62-003d291e2b14" containerName="revoke-keys" Oct 31 13:42:12.822919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.822900 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.828458 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.828422 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:42:12.828584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.828466 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:42:12.829817 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.829783 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26"] Oct 31 13:42:12.869691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.868791 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:42:12.869691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.868867 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:42:12.892206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.892176 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjnj\" (UniqueName: \"kubernetes.io/projected/50b73787-fef7-41d2-8105-feb5dcf6805a-kube-api-access-ghjnj\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.892370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.892224 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.892457 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.892403 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.994392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.994351 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.994603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.994485 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjnj\" (UniqueName: \"kubernetes.io/projected/50b73787-fef7-41d2-8105-feb5dcf6805a-kube-api-access-ghjnj\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.994603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.994537 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.995227 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.995202 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:12.997058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:12.997029 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:13.007078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:13.007055 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjnj\" (UniqueName: \"kubernetes.io/projected/50b73787-fef7-41d2-8105-feb5dcf6805a-kube-api-access-ghjnj\") pod \"maas-api-revoke-sub-adeac515-d7z26\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:13.138726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:13.138626 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:13.425122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:13.425087 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26"] Oct 31 13:42:13.425551 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:13.425527 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b73787_fef7_41d2_8105_feb5dcf6805a.slice/crio-4819eeaab7f786dfd7ce6b603056d1f76df8ecb9ff63373c18f92f229ded9dad WatchSource:0}: Error finding container 4819eeaab7f786dfd7ce6b603056d1f76df8ecb9ff63373c18f92f229ded9dad: Status 404 returned error can't find the container with id 4819eeaab7f786dfd7ce6b603056d1f76df8ecb9ff63373c18f92f229ded9dad Oct 31 13:42:13.917194 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:13.917146 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" event={"ID":"50b73787-fef7-41d2-8105-feb5dcf6805a","Type":"ContainerStarted","Data":"bfbc5f9a2b8fceba47cfb3c3440dbe8f3fc0d9210de9fbd1e734ef94a407e8aa"} Oct 31 13:42:13.917605 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:13.917218 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" event={"ID":"50b73787-fef7-41d2-8105-feb5dcf6805a","Type":"ContainerStarted","Data":"4819eeaab7f786dfd7ce6b603056d1f76df8ecb9ff63373c18f92f229ded9dad"} Oct 31 13:42:13.939153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:13.939047 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" podStartSLOduration=1.939032951 podStartE2EDuration="1.939032951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:13.935711194 +0000 UTC m=+4228.113429179" watchObservedRunningTime="2026-10-31 13:42:13.939032951 +0000 UTC m=+4228.116750935" Oct 31 13:42:14.733839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.733803 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt"] Oct 31 13:42:14.738093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.738074 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.748281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.748258 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt"] Oct 31 13:42:14.819390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.819356 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.819390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.819397 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmrv\" (UniqueName: \"kubernetes.io/projected/f87d871b-7840-429c-a655-0b8be93f4616-kube-api-access-ghmrv\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.819711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.819671 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.920744 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.920696 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.921178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.920887 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.921178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.920919 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmrv\" (UniqueName: \"kubernetes.io/projected/f87d871b-7840-429c-a655-0b8be93f4616-kube-api-access-ghmrv\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.921497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.921471 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.923508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.923480 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:14.926721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.926696 5074 generic.go:356] "Generic (PLEG): container finished" podID="50b73787-fef7-41d2-8105-feb5dcf6805a" containerID="bfbc5f9a2b8fceba47cfb3c3440dbe8f3fc0d9210de9fbd1e734ef94a407e8aa" exitCode=0 Oct 31 13:42:14.926820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.926772 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" event={"ID":"50b73787-fef7-41d2-8105-feb5dcf6805a","Type":"ContainerDied","Data":"bfbc5f9a2b8fceba47cfb3c3440dbe8f3fc0d9210de9fbd1e734ef94a407e8aa"} Oct 31 13:42:14.934101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:14.934076 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmrv\" (UniqueName: \"kubernetes.io/projected/f87d871b-7840-429c-a655-0b8be93f4616-kube-api-access-ghmrv\") pod \"maas-api-revoke-sub-bfd6b27f-np5nt\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:15.054908 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:15.054829 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:15.356894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:15.356866 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt"] Oct 31 13:42:15.357123 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:15.357099 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87d871b_7840_429c_a655_0b8be93f4616.slice/crio-4d9077607db74d67a961b12c94efb1d70ac4be06f89853d1c125b8ac42edbb70 WatchSource:0}: Error finding container 4d9077607db74d67a961b12c94efb1d70ac4be06f89853d1c125b8ac42edbb70: Status 404 returned error can't find the container with id 4d9077607db74d67a961b12c94efb1d70ac4be06f89853d1c125b8ac42edbb70 Oct 31 13:42:15.939760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:15.939728 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" event={"ID":"f87d871b-7840-429c-a655-0b8be93f4616","Type":"ContainerStarted","Data":"2684f13550f75bec98185b432648ccb9ae85fca2ce16021964cb59a2e7e3730b"} Oct 31 13:42:15.940136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:15.939779 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" event={"ID":"f87d871b-7840-429c-a655-0b8be93f4616","Type":"ContainerStarted","Data":"4d9077607db74d67a961b12c94efb1d70ac4be06f89853d1c125b8ac42edbb70"} Oct 31 13:42:15.962106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:15.962051 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" podStartSLOduration=1.962036666 podStartE2EDuration="1.962036666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:15.95998754 +0000 UTC m=+4230.137705523" watchObservedRunningTime="2026-10-31 13:42:15.962036666 +0000 UTC m=+4230.139754686" Oct 31 13:42:16.164675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.164617 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:16.239533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.239499 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b73787-fef7-41d2-8105-feb5dcf6805a-kube-api-access-ghjnj\" (UniqueName: \"kubernetes.io/projected/50b73787-fef7-41d2-8105-feb5dcf6805a-kube-api-access-ghjnj\") pod \"50b73787-fef7-41d2-8105-feb5dcf6805a\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " Oct 31 13:42:16.239733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.239568 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-xks-ca\") pod \"50b73787-fef7-41d2-8105-feb5dcf6805a\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " Oct 31 13:42:16.239733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.239605 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-openshift-ca\") pod \"50b73787-fef7-41d2-8105-feb5dcf6805a\" (UID: \"50b73787-fef7-41d2-8105-feb5dcf6805a\") " Oct 31 13:42:16.240095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.240057 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-openshift-ca" pod "50b73787-fef7-41d2-8105-feb5dcf6805a" (UID: "50b73787-fef7-41d2-8105-feb5dcf6805a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:16.240394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.240374 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:16.241768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.241736 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-xks-ca" pod "50b73787-fef7-41d2-8105-feb5dcf6805a" (UID: "50b73787-fef7-41d2-8105-feb5dcf6805a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:16.241885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.241745 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b73787-fef7-41d2-8105-feb5dcf6805a-kube-api-access-ghjnj" pod "50b73787-fef7-41d2-8105-feb5dcf6805a" (UID: "50b73787-fef7-41d2-8105-feb5dcf6805a"). InnerVolumeSpecName "kube-api-access-ghjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:16.341292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.341257 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghjnj\" (UniqueName: \"kubernetes.io/projected/50b73787-fef7-41d2-8105-feb5dcf6805a-kube-api-access-ghjnj\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:16.341292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.341289 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b73787-fef7-41d2-8105-feb5dcf6805a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:16.946133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.946092 5074 generic.go:356] "Generic (PLEG): container finished" podID="f87d871b-7840-429c-a655-0b8be93f4616" containerID="2684f13550f75bec98185b432648ccb9ae85fca2ce16021964cb59a2e7e3730b" exitCode=0 Oct 31 13:42:16.946619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.946160 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" event={"ID":"f87d871b-7840-429c-a655-0b8be93f4616","Type":"ContainerDied","Data":"2684f13550f75bec98185b432648ccb9ae85fca2ce16021964cb59a2e7e3730b"} Oct 31 13:42:16.947827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.947795 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" Oct 31 13:42:16.948011 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.947850 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26" event={"ID":"50b73787-fef7-41d2-8105-feb5dcf6805a","Type":"ContainerDied","Data":"4819eeaab7f786dfd7ce6b603056d1f76df8ecb9ff63373c18f92f229ded9dad"} Oct 31 13:42:16.948011 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:16.947900 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4819eeaab7f786dfd7ce6b603056d1f76df8ecb9ff63373c18f92f229ded9dad" Oct 31 13:42:18.137624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.137578 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:18.296999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.296955 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-xks-ca\") pod \"f87d871b-7840-429c-a655-0b8be93f4616\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " Oct 31 13:42:18.297214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.297014 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-openshift-ca\") pod \"f87d871b-7840-429c-a655-0b8be93f4616\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " Oct 31 13:42:18.297214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.297149 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87d871b-7840-429c-a655-0b8be93f4616-kube-api-access-ghmrv\" (UniqueName: \"kubernetes.io/projected/f87d871b-7840-429c-a655-0b8be93f4616-kube-api-access-ghmrv\") pod \"f87d871b-7840-429c-a655-0b8be93f4616\" (UID: \"f87d871b-7840-429c-a655-0b8be93f4616\") " Oct 31 13:42:18.297469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.297439 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-openshift-ca" pod "f87d871b-7840-429c-a655-0b8be93f4616" (UID: "f87d871b-7840-429c-a655-0b8be93f4616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:18.297917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.297891 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:18.299215 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.299188 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-xks-ca" pod "f87d871b-7840-429c-a655-0b8be93f4616" (UID: "f87d871b-7840-429c-a655-0b8be93f4616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:18.299373 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.299354 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87d871b-7840-429c-a655-0b8be93f4616-kube-api-access-ghmrv" pod "f87d871b-7840-429c-a655-0b8be93f4616" (UID: "f87d871b-7840-429c-a655-0b8be93f4616"). InnerVolumeSpecName "kube-api-access-ghmrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:18.399579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.399536 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87d871b-7840-429c-a655-0b8be93f4616-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:18.399579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.399576 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmrv\" (UniqueName: \"kubernetes.io/projected/f87d871b-7840-429c-a655-0b8be93f4616-kube-api-access-ghmrv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:18.963860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.963827 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" Oct 31 13:42:18.963860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.963842 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt" event={"ID":"f87d871b-7840-429c-a655-0b8be93f4616","Type":"ContainerDied","Data":"4d9077607db74d67a961b12c94efb1d70ac4be06f89853d1c125b8ac42edbb70"} Oct 31 13:42:18.964127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:18.963901 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9077607db74d67a961b12c94efb1d70ac4be06f89853d1c125b8ac42edbb70" Oct 31 13:42:21.079642 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:21.079595 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx"] Oct 31 13:42:21.104938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:21.104909 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275549df-5p8wx"] Oct 31 13:42:21.941391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:21.941335 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0bc38b-efce-493f-aa8b-4c4c28181384" path="/var/lib/kubelet/pods/6c0bc38b-efce-493f-aa8b-4c4c28181384/volumes" Oct 31 13:42:23.427766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.427721 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk"] Oct 31 13:42:23.428722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.428700 5074 cpu_manager.go:414] "containerMap: removing container" podUID="50b73787-fef7-41d2-8105-feb5dcf6805a" containerName="revoke-keys" Oct 31 13:42:23.428722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.428722 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b73787-fef7-41d2-8105-feb5dcf6805a" containerName="revoke-keys" Oct 31 13:42:23.428907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.428774 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f87d871b-7840-429c-a655-0b8be93f4616" containerName="revoke-keys" Oct 31 13:42:23.428907 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.428782 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87d871b-7840-429c-a655-0b8be93f4616" containerName="revoke-keys" Oct 31 13:42:23.429035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.428923 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b73787-fef7-41d2-8105-feb5dcf6805a" containerName="revoke-keys" Oct 31 13:42:23.429035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.428943 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87d871b-7840-429c-a655-0b8be93f4616" containerName="revoke-keys" Oct 31 13:42:23.442838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.438040 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.447611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.447529 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk"] Oct 31 13:42:23.458692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.456421 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:42:23.458692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.456544 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:42:23.466775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.461195 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:42:23.466775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.461354 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:42:23.491254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.491198 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.491469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.491352 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.491551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.491504 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pnwv\" (UniqueName: \"kubernetes.io/projected/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-kube-api-access-4pnwv\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.606715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.602894 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pnwv\" (UniqueName: \"kubernetes.io/projected/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-kube-api-access-4pnwv\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.606715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.603020 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.606715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.603133 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.606715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.604824 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.666686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.661164 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.680731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.679486 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pnwv\" (UniqueName: \"kubernetes.io/projected/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-kube-api-access-4pnwv\") pod \"maas-api-revoke-sub-8ab9bb5c-mxngk\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:23.765110 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:23.765051 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:24.058613 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:24.058171 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk"] Oct 31 13:42:24.058926 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:24.058890 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ac7bc_ce3c_4771_a315_3e2fe96ebf7a.slice/crio-856c80f2b3ed9ab750032ebbbcd923edaac0b3462887952a616ae7777d975ae1 WatchSource:0}: Error finding container 856c80f2b3ed9ab750032ebbbcd923edaac0b3462887952a616ae7777d975ae1: Status 404 returned error can't find the container with id 856c80f2b3ed9ab750032ebbbcd923edaac0b3462887952a616ae7777d975ae1 Oct 31 13:42:25.052526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:25.052492 5074 generic.go:356] "Generic (PLEG): container finished" podID="cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" containerID="cacc2ba08dd0a44b8857ce9707ed98ce92f857b7bb39866f4a5b7a5405db7713" exitCode=0 Oct 31 13:42:25.052983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:25.052542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" event={"ID":"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a","Type":"ContainerDied","Data":"cacc2ba08dd0a44b8857ce9707ed98ce92f857b7bb39866f4a5b7a5405db7713"} Oct 31 13:42:25.052983 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:25.052597 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" event={"ID":"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a","Type":"ContainerStarted","Data":"856c80f2b3ed9ab750032ebbbcd923edaac0b3462887952a616ae7777d975ae1"} Oct 31 13:42:25.985374 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:42:25.985331 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:42:26.244595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.244520 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:26.347570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.347528 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-kube-api-access-4pnwv\" (UniqueName: \"kubernetes.io/projected/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-kube-api-access-4pnwv\") pod \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " Oct 31 13:42:26.347800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.347588 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-openshift-ca\") pod \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " Oct 31 13:42:26.347800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.347747 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-xks-ca\") pod \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\" (UID: \"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a\") " Oct 31 13:42:26.348042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.348010 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-openshift-ca" pod "cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" (UID: "cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:26.348282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.348261 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:26.349902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.349870 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-xks-ca" pod "cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" (UID: "cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:26.350033 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.350010 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-kube-api-access-4pnwv" pod "cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" (UID: "cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a"). InnerVolumeSpecName "kube-api-access-4pnwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:26.449697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.449664 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pnwv\" (UniqueName: \"kubernetes.io/projected/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-kube-api-access-4pnwv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:26.449697 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:26.449694 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:27.070909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:27.070858 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" event={"ID":"cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a","Type":"ContainerDied","Data":"856c80f2b3ed9ab750032ebbbcd923edaac0b3462887952a616ae7777d975ae1"} Oct 31 13:42:27.071115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:27.070939 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="856c80f2b3ed9ab750032ebbbcd923edaac0b3462887952a616ae7777d975ae1" Oct 31 13:42:27.071115 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:27.070878 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk" Oct 31 13:42:30.381471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:30.381431 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92"] Oct 31 13:42:30.962051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:30.962009 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5"] Oct 31 13:42:30.962286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:30.962260 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" podUID="070fa1fe-3a61-4c63-8566-c87f4d999aff" containerName="discovery" containerID="cri-o://feac327767c27bc8d95df2da0a205fc954f51821e0ad64700ad420db5d0d0229" gracePeriod=30 Oct 31 13:42:31.139192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.139148 5074 generic.go:356] "Generic (PLEG): container finished" podID="070fa1fe-3a61-4c63-8566-c87f4d999aff" containerID="feac327767c27bc8d95df2da0a205fc954f51821e0ad64700ad420db5d0d0229" exitCode=0 Oct 31 13:42:31.142307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.142273 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" event={"ID":"070fa1fe-3a61-4c63-8566-c87f4d999aff","Type":"ContainerDied","Data":"feac327767c27bc8d95df2da0a205fc954f51821e0ad64700ad420db5d0d0229"} Oct 31 13:42:31.287969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.287940 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 13:42:31.347944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.347905 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070fa1fe-3a61-4c63-8566-c87f4d999aff-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/070fa1fe-3a61-4c63-8566-c87f4d999aff-local-certs\") pod \"070fa1fe-3a61-4c63-8566-c87f4d999aff\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " Oct 31 13:42:31.348147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.347992 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-cacerts\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-cacerts\") pod \"070fa1fe-3a61-4c63-8566-c87f4d999aff\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " Oct 31 13:42:31.348147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.348035 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-ca-configmap\") pod \"070fa1fe-3a61-4c63-8566-c87f4d999aff\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " Oct 31 13:42:31.348147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.348060 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-kube-api-access-j52n2\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-kube-api-access-j52n2\") pod \"070fa1fe-3a61-4c63-8566-c87f4d999aff\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " Oct 31 13:42:31.348331 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.348192 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-kubeconfig\") pod \"070fa1fe-3a61-4c63-8566-c87f4d999aff\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " Oct 31 13:42:31.348415 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.348394 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-token\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-token\") pod \"070fa1fe-3a61-4c63-8566-c87f4d999aff\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " Oct 31 13:42:31.348520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.348498 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-ca-configmap" pod "070fa1fe-3a61-4c63-8566-c87f4d999aff" (UID: "070fa1fe-3a61-4c63-8566-c87f4d999aff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:31.348520 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.348511 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-dns-cert\") pod \"070fa1fe-3a61-4c63-8566-c87f4d999aff\" (UID: \"070fa1fe-3a61-4c63-8566-c87f4d999aff\") " Oct 31 13:42:31.349200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.349171 5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-ca-configmap\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:31.351077 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.351045 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070fa1fe-3a61-4c63-8566-c87f4d999aff-local-certs" pod "070fa1fe-3a61-4c63-8566-c87f4d999aff" (UID: "070fa1fe-3a61-4c63-8566-c87f4d999aff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:31.351210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.351096 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-token" pod "070fa1fe-3a61-4c63-8566-c87f4d999aff" (UID: "070fa1fe-3a61-4c63-8566-c87f4d999aff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:31.351210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.351177 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-kubeconfig" pod "070fa1fe-3a61-4c63-8566-c87f4d999aff" (UID: "070fa1fe-3a61-4c63-8566-c87f4d999aff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:31.351326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.351299 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-kube-api-access-j52n2" pod "070fa1fe-3a61-4c63-8566-c87f4d999aff" (UID: "070fa1fe-3a61-4c63-8566-c87f4d999aff"). InnerVolumeSpecName "kube-api-access-j52n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:31.351498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.351479 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-cacerts" pod "070fa1fe-3a61-4c63-8566-c87f4d999aff" (UID: "070fa1fe-3a61-4c63-8566-c87f4d999aff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:31.351759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.351730 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-dns-cert" pod "070fa1fe-3a61-4c63-8566-c87f4d999aff" (UID: "070fa1fe-3a61-4c63-8566-c87f4d999aff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:31.450271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.450234 5074 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-kubeconfig\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:31.450271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.450264 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:31.450271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.450274 5074 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-istio-csr-dns-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:31.450802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.450287 5074 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/070fa1fe-3a61-4c63-8566-c87f4d999aff-local-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:31.450802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.450296 5074 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/070fa1fe-3a61-4c63-8566-c87f4d999aff-cacerts\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:31.450802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:31.450306 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52n2\" (UniqueName: \"kubernetes.io/projected/070fa1fe-3a61-4c63-8566-c87f4d999aff-kube-api-access-j52n2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:32.154076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:32.154044 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" Oct 31 13:42:32.154263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:32.154071 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5" event={"ID":"070fa1fe-3a61-4c63-8566-c87f4d999aff","Type":"ContainerDied","Data":"185e902d621bd902205440fd4a9e9c02671ef67c0fa1c7aed86494bcc213b170"} Oct 31 13:42:32.154263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:32.154136 5074 scope.go:122] "RemoveContainer" containerID="feac327767c27bc8d95df2da0a205fc954f51821e0ad64700ad420db5d0d0229" Oct 31 13:42:32.180698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:32.180625 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5"] Oct 31 13:42:32.184091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:32.184061 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7lb5"] Oct 31 13:42:33.066435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:33.066400 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd"] Oct 31 13:42:33.070065 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:33.070031 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9de603-wqvpd"] Oct 31 13:42:33.940693 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:33.940658 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070fa1fe-3a61-4c63-8566-c87f4d999aff" path="/var/lib/kubelet/pods/070fa1fe-3a61-4c63-8566-c87f4d999aff/volumes" Oct 31 13:42:33.941197 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:33.941181 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96340e4f-525f-4547-8f4a-64e1ad317fbb" path="/var/lib/kubelet/pods/96340e4f-525f-4547-8f4a-64e1ad317fbb/volumes" Oct 31 13:42:35.381938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:35.381869 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" podUID="47db0401-e110-4eb4-ac8a-37b7c864191e" containerName="payload-processing" containerID="cri-o://82097e6bb854595822a2abdfe4d04c53c301c6366b8b117f6a442286010b1658" gracePeriod=25 Oct 31 13:42:41.018936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:41.018883 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" podUID="47db0401-e110-4eb4-ac8a-37b7c864191e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:46.328556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:46.328506 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:42:47.198392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:47.198354 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9"] Oct 31 13:42:47.198616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:47.198588 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" containerID="cri-o://64f930168240967cc60f1dbba9b2372c71a46c6921926a7ed40ecf9c15c117f5" gracePeriod=30 Oct 31 13:42:48.095089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.095047 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9"] Oct 31 13:42:48.095686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.095667 5074 cpu_manager.go:414] "containerMap: removing container" podUID="070fa1fe-3a61-4c63-8566-c87f4d999aff" containerName="discovery" Oct 31 13:42:48.095686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.095684 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070fa1fe-3a61-4c63-8566-c87f4d999aff" containerName="discovery" Oct 31 13:42:48.095814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.095726 5074 cpu_manager.go:414] "containerMap: removing container" podUID="cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" containerName="revoke-keys" Oct 31 13:42:48.095814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.095733 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" containerName="revoke-keys" Oct 31 13:42:48.095887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.095833 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" containerName="revoke-keys" Oct 31 13:42:48.095887 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.095845 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="070fa1fe-3a61-4c63-8566-c87f4d999aff" containerName="discovery" Oct 31 13:42:48.099624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.099606 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.112142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.112117 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:42:48.112329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.112280 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:42:48.114208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.114184 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:42:48.114208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.114198 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:42:48.116417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.116393 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9"] Oct 31 13:42:48.173856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.173817 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.174048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.174029 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.174095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.174074 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstzp\" (UniqueName: \"kubernetes.io/projected/6783d96d-5e10-4760-9dd8-a79d495ff706-kube-api-access-vstzp\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.274971 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.274936 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.275152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.275112 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.275220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.275160 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstzp\" (UniqueName: \"kubernetes.io/projected/6783d96d-5e10-4760-9dd8-a79d495ff706-kube-api-access-vstzp\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.275968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.275940 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.277608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.277573 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.288084 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.288050 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstzp\" (UniqueName: \"kubernetes.io/projected/6783d96d-5e10-4760-9dd8-a79d495ff706-kube-api-access-vstzp\") pod \"maas-api-revoke-sub-65d5cf47-6xzr9\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.414018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.413920 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:48.742483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:48.742452 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9"] Oct 31 13:42:48.743738 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:48.743706 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6783d96d_5e10_4760_9dd8_a79d495ff706.slice/crio-4256996728c816c304ecb5a4971f86678c192ca535d73c03b22ce0c95f84d7b3 WatchSource:0}: Error finding container 4256996728c816c304ecb5a4971f86678c192ca535d73c03b22ce0c95f84d7b3: Status 404 returned error can't find the container with id 4256996728c816c304ecb5a4971f86678c192ca535d73c03b22ce0c95f84d7b3 Oct 31 13:42:49.293438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:49.293400 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" event={"ID":"6783d96d-5e10-4760-9dd8-a79d495ff706","Type":"ContainerStarted","Data":"36d3eca70e139d77f7ca4ab8d45b2bd5362a17b5a9e5529eb3fd33de765b2115"} Oct 31 13:42:49.293940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:49.293466 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" event={"ID":"6783d96d-5e10-4760-9dd8-a79d495ff706","Type":"ContainerStarted","Data":"4256996728c816c304ecb5a4971f86678c192ca535d73c03b22ce0c95f84d7b3"} Oct 31 13:42:49.319375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:49.319316 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" podStartSLOduration=1.319303175 podStartE2EDuration="1.319303175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:49.316166685 +0000 UTC m=+4263.493884672" watchObservedRunningTime="2026-10-31 13:42:49.319303175 +0000 UTC m=+4263.497021159" Oct 31 13:42:49.983845 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:49.983809 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn"] Oct 31 13:42:49.988147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:49.988124 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:49.998665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:49.998491 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn"] Oct 31 13:42:50.104022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.103978 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br5fn\" (UniqueName: \"kubernetes.io/projected/3560fe0a-a54a-4a6c-8df4-a47145c602ed-kube-api-access-br5fn\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.104243 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.104211 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.104425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.104406 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.205972 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.205921 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.206184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.206019 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br5fn\" (UniqueName: \"kubernetes.io/projected/3560fe0a-a54a-4a6c-8df4-a47145c602ed-kube-api-access-br5fn\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.206184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.206152 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.206670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.206626 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.208597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.208562 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.218556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.218530 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br5fn\" (UniqueName: \"kubernetes.io/projected/3560fe0a-a54a-4a6c-8df4-a47145c602ed-kube-api-access-br5fn\") pod \"maas-api-revoke-sub-86efb512-w6dpn\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.305594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.305551 5074 generic.go:356] "Generic (PLEG): container finished" podID="6783d96d-5e10-4760-9dd8-a79d495ff706" containerID="36d3eca70e139d77f7ca4ab8d45b2bd5362a17b5a9e5529eb3fd33de765b2115" exitCode=0 Oct 31 13:42:50.306032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.305594 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" event={"ID":"6783d96d-5e10-4760-9dd8-a79d495ff706","Type":"ContainerDied","Data":"36d3eca70e139d77f7ca4ab8d45b2bd5362a17b5a9e5529eb3fd33de765b2115"} Oct 31 13:42:50.313267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.313240 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:50.615067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:50.615033 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn"] Oct 31 13:42:50.615607 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:50.615566 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3560fe0a_a54a_4a6c_8df4_a47145c602ed.slice/crio-014c1762e80cb2229943c1da65d81f752840d9d9bbb6203646bea632f624909d WatchSource:0}: Error finding container 014c1762e80cb2229943c1da65d81f752840d9d9bbb6203646bea632f624909d: Status 404 returned error can't find the container with id 014c1762e80cb2229943c1da65d81f752840d9d9bbb6203646bea632f624909d Oct 31 13:42:51.019619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.019585 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" podUID="47db0401-e110-4eb4-ac8a-37b7c864191e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:42:51.076800 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.076748 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:42:51.077144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.077072 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:42:51.317513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.317406 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" event={"ID":"3560fe0a-a54a-4a6c-8df4-a47145c602ed","Type":"ContainerStarted","Data":"29377f311874b25278c70d95ffb218eef59a028c9ed523021fc6a1168f1a5910"} Oct 31 13:42:51.317513 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.317476 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" event={"ID":"3560fe0a-a54a-4a6c-8df4-a47145c602ed","Type":"ContainerStarted","Data":"014c1762e80cb2229943c1da65d81f752840d9d9bbb6203646bea632f624909d"} Oct 31 13:42:51.340049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.339996 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" podStartSLOduration=2.339982822 podStartE2EDuration="2.339982822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:51.336441354 +0000 UTC m=+4265.514159340" watchObservedRunningTime="2026-10-31 13:42:51.339982822 +0000 UTC m=+4265.517700803" Oct 31 13:42:51.494794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.494761 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:51.627627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.627593 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-openshift-ca\") pod \"6783d96d-5e10-4760-9dd8-a79d495ff706\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " Oct 31 13:42:51.627811 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.627750 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6783d96d-5e10-4760-9dd8-a79d495ff706-kube-api-access-vstzp\" (UniqueName: \"kubernetes.io/projected/6783d96d-5e10-4760-9dd8-a79d495ff706-kube-api-access-vstzp\") pod \"6783d96d-5e10-4760-9dd8-a79d495ff706\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " Oct 31 13:42:51.627992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.627962 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-xks-ca\") pod \"6783d96d-5e10-4760-9dd8-a79d495ff706\" (UID: \"6783d96d-5e10-4760-9dd8-a79d495ff706\") " Oct 31 13:42:51.628199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.628004 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-openshift-ca" pod "6783d96d-5e10-4760-9dd8-a79d495ff706" (UID: "6783d96d-5e10-4760-9dd8-a79d495ff706"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:51.628450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.628427 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:51.630018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.629990 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6783d96d-5e10-4760-9dd8-a79d495ff706-kube-api-access-vstzp" pod "6783d96d-5e10-4760-9dd8-a79d495ff706" (UID: "6783d96d-5e10-4760-9dd8-a79d495ff706"). InnerVolumeSpecName "kube-api-access-vstzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:51.630102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.630059 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-xks-ca" pod "6783d96d-5e10-4760-9dd8-a79d495ff706" (UID: "6783d96d-5e10-4760-9dd8-a79d495ff706"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:51.730355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.730314 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstzp\" (UniqueName: \"kubernetes.io/projected/6783d96d-5e10-4760-9dd8-a79d495ff706-kube-api-access-vstzp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:51.730355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:51.730350 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6783d96d-5e10-4760-9dd8-a79d495ff706-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.327397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.327367 5074 generic.go:356] "Generic (PLEG): container finished" podID="3560fe0a-a54a-4a6c-8df4-a47145c602ed" containerID="29377f311874b25278c70d95ffb218eef59a028c9ed523021fc6a1168f1a5910" exitCode=0 Oct 31 13:42:52.327829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.327451 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" event={"ID":"3560fe0a-a54a-4a6c-8df4-a47145c602ed","Type":"ContainerDied","Data":"29377f311874b25278c70d95ffb218eef59a028c9ed523021fc6a1168f1a5910"} Oct 31 13:42:52.329779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.329754 5074 generic.go:356] "Generic (PLEG): container finished" podID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerID="64f930168240967cc60f1dbba9b2372c71a46c6921926a7ed40ecf9c15c117f5" exitCode=0 Oct 31 13:42:52.329881 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.329836 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" event={"ID":"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27","Type":"ContainerDied","Data":"64f930168240967cc60f1dbba9b2372c71a46c6921926a7ed40ecf9c15c117f5"} Oct 31 13:42:52.329963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.329902 5074 scope.go:122] "RemoveContainer" containerID="096ac549e26d41832da50adde2f199f3d5e1c62cad6a84f3216624b0db57b9b4" Oct 31 13:42:52.331873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.331851 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" Oct 31 13:42:52.331978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.331885 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9" event={"ID":"6783d96d-5e10-4760-9dd8-a79d495ff706","Type":"ContainerDied","Data":"4256996728c816c304ecb5a4971f86678c192ca535d73c03b22ce0c95f84d7b3"} Oct 31 13:42:52.331978 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.331927 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4256996728c816c304ecb5a4971f86678c192ca535d73c03b22ce0c95f84d7b3" Oct 31 13:42:52.522372 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.522337 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:42:52.620046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.620010 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb"] Oct 31 13:42:52.625178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.625144 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7"] Oct 31 13:42:52.634586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.634554 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg"] Oct 31 13:42:52.658827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.658797 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-envoy\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.658834 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-credential-socket\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.658918 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-kube-api-access-2nd75\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-kube-api-access-2nd75\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.658956 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istiod-ca-cert\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659046 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-podinfo\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659098 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-certs\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659223 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-data\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659234 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-credential-socket" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:52.659364 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659290 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-socket\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659390 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-certs" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:52.659545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659403 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-token\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-token\") pod \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\" (UID: \"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27\") " Oct 31 13:42:52.659545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659459 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-data" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:52.659752 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.659645 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-socket" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:52.660152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.660121 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.660259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.660186 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.660259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.660205 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.660259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.660221 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.660434 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.660335 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istiod-ca-cert" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:52.661436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.661410 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-kube-api-access-2nd75" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "kube-api-access-2nd75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:52.661587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.661563 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-envoy" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:52.662217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.662194 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-token" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:52.662411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.662395 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-podinfo" pod "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" (UID: "a9d13e1c-c11e-4d70-8c1b-01c2679a2f27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:42:52.761816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.761726 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.761816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.761760 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.761816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.761773 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.761816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.761788 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nd75\" (UniqueName: \"kubernetes.io/projected/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-kube-api-access-2nd75\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:52.761816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:52.761803 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:53.340362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.340327 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" Oct 31 13:42:53.340792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.340337 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9" event={"ID":"a9d13e1c-c11e-4d70-8c1b-01c2679a2f27","Type":"ContainerDied","Data":"eb4f97919bdedf2fd98e4baf70b3dd49247b58db6afc6ce74caf651b881a0b3a"} Oct 31 13:42:53.340792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.340492 5074 scope.go:122] "RemoveContainer" containerID="64f930168240967cc60f1dbba9b2372c71a46c6921926a7ed40ecf9c15c117f5" Oct 31 13:42:53.379937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.379872 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9"] Oct 31 13:42:53.385966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.385923 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c24e1-openshift-default-6db858867f-lsnv9"] Oct 31 13:42:53.503835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.503805 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:53.575562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.575522 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-xks-ca\") pod \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " Oct 31 13:42:53.575819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.575598 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3560fe0a-a54a-4a6c-8df4-a47145c602ed-kube-api-access-br5fn\" (UniqueName: \"kubernetes.io/projected/3560fe0a-a54a-4a6c-8df4-a47145c602ed-kube-api-access-br5fn\") pod \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " Oct 31 13:42:53.576132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.576099 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-openshift-ca\") pod \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\" (UID: \"3560fe0a-a54a-4a6c-8df4-a47145c602ed\") " Oct 31 13:42:53.576510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.576478 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-openshift-ca" pod "3560fe0a-a54a-4a6c-8df4-a47145c602ed" (UID: "3560fe0a-a54a-4a6c-8df4-a47145c602ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:53.576883 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.576861 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:53.588691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.579085 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-xks-ca" pod "3560fe0a-a54a-4a6c-8df4-a47145c602ed" (UID: "3560fe0a-a54a-4a6c-8df4-a47145c602ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:42:53.588691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.581057 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3560fe0a-a54a-4a6c-8df4-a47145c602ed-kube-api-access-br5fn" pod "3560fe0a-a54a-4a6c-8df4-a47145c602ed" (UID: "3560fe0a-a54a-4a6c-8df4-a47145c602ed"). InnerVolumeSpecName "kube-api-access-br5fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:53.679043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.678953 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3560fe0a-a54a-4a6c-8df4-a47145c602ed-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:53.679043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.678999 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br5fn\" (UniqueName: \"kubernetes.io/projected/3560fe0a-a54a-4a6c-8df4-a47145c602ed-kube-api-access-br5fn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:53.939955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:53.939876 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" path="/var/lib/kubelet/pods/a9d13e1c-c11e-4d70-8c1b-01c2679a2f27/volumes" Oct 31 13:42:54.354648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:54.354608 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" Oct 31 13:42:54.355230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:54.354625 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn" event={"ID":"3560fe0a-a54a-4a6c-8df4-a47145c602ed","Type":"ContainerDied","Data":"014c1762e80cb2229943c1da65d81f752840d9d9bbb6203646bea632f624909d"} Oct 31 13:42:54.355230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:54.354713 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="014c1762e80cb2229943c1da65d81f752840d9d9bbb6203646bea632f624909d" Oct 31 13:42:54.713381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:54.713287 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2"] Oct 31 13:42:55.688979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:55.688949 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:42:55.815363 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:55.815320 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47db0401-e110-4eb4-ac8a-37b7c864191e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47db0401-e110-4eb4-ac8a-37b7c864191e-plugins-config-volume\") pod \"47db0401-e110-4eb4-ac8a-37b7c864191e\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " Oct 31 13:42:55.815623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:55.815602 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47db0401-e110-4eb4-ac8a-37b7c864191e-kube-api-access-5s6q6\" (UniqueName: \"kubernetes.io/projected/47db0401-e110-4eb4-ac8a-37b7c864191e-kube-api-access-5s6q6\") pod \"47db0401-e110-4eb4-ac8a-37b7c864191e\" (UID: \"47db0401-e110-4eb4-ac8a-37b7c864191e\") " Oct 31 13:42:55.815795 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:55.815765 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47db0401-e110-4eb4-ac8a-37b7c864191e-plugins-config-volume" pod "47db0401-e110-4eb4-ac8a-37b7c864191e" (UID: "47db0401-e110-4eb4-ac8a-37b7c864191e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:55.816251 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:55.816221 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47db0401-e110-4eb4-ac8a-37b7c864191e-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:55.817856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:55.817829 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47db0401-e110-4eb4-ac8a-37b7c864191e-kube-api-access-5s6q6" pod "47db0401-e110-4eb4-ac8a-37b7c864191e" (UID: "47db0401-e110-4eb4-ac8a-37b7c864191e"). InnerVolumeSpecName "kube-api-access-5s6q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:55.917703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:55.917604 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s6q6\" (UniqueName: \"kubernetes.io/projected/47db0401-e110-4eb4-ac8a-37b7c864191e-kube-api-access-5s6q6\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:55.991274 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:42:55.991230 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:42:56.309109 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.309039 5074 scope.go:122] "RemoveContainer" containerID="3f1ad1295de7d738c4480c0942d664b0189fa45b902e1ea2793ebb379f10be04" Oct 31 13:42:56.334929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.332144 5074 scope.go:122] "RemoveContainer" containerID="91aa1739ab27008b4477f260cf42521370024f9849ca2f8419a250d1da67a512" Oct 31 13:42:56.352329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.352301 5074 scope.go:122] "RemoveContainer" containerID="0afe02d4cb78ad9ffd527b63c53336fb925bc676ba68eb502577081caee96dcc" Oct 31 13:42:56.362255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.362230 5074 scope.go:122] "RemoveContainer" containerID="82097e6bb854595822a2abdfe4d04c53c301c6366b8b117f6a442286010b1658" Oct 31 13:42:56.375966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.375206 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" Oct 31 13:42:56.375966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.375215 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" event={"ID":"47db0401-e110-4eb4-ac8a-37b7c864191e","Type":"ContainerDied","Data":"82097e6bb854595822a2abdfe4d04c53c301c6366b8b117f6a442286010b1658"} Oct 31 13:42:56.375966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.375282 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92" event={"ID":"47db0401-e110-4eb4-ac8a-37b7c864191e","Type":"ContainerDied","Data":"cf303a34e6f221ffdcb98e6792aea6a4378fe39039b0565bc510543270e68eb8"} Oct 31 13:42:56.408955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.408918 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92"] Oct 31 13:42:56.409105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:56.408982 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c24e1-b4765668f-fvw92"] Oct 31 13:42:57.941890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:57.941853 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47db0401-e110-4eb4-ac8a-37b7c864191e" path="/var/lib/kubelet/pods/47db0401-e110-4eb4-ac8a-37b7c864191e/volumes" Oct 31 13:42:58.207475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.207396 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck"] Oct 31 13:42:58.208106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208083 5074 cpu_manager.go:414] "containerMap: removing container" podUID="3560fe0a-a54a-4a6c-8df4-a47145c602ed" containerName="revoke-keys" Oct 31 13:42:58.208106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208105 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3560fe0a-a54a-4a6c-8df4-a47145c602ed" containerName="revoke-keys" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208127 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208133 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208146 5074 cpu_manager.go:414] "containerMap: removing container" podUID="47db0401-e110-4eb4-ac8a-37b7c864191e" containerName="payload-processing" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208152 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47db0401-e110-4eb4-ac8a-37b7c864191e" containerName="payload-processing" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208166 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6783d96d-5e10-4760-9dd8-a79d495ff706" containerName="revoke-keys" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208171 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6783d96d-5e10-4760-9dd8-a79d495ff706" containerName="revoke-keys" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208181 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" Oct 31 13:42:58.208261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208186 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" Oct 31 13:42:58.208582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208283 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="3560fe0a-a54a-4a6c-8df4-a47145c602ed" containerName="revoke-keys" Oct 31 13:42:58.208582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208292 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" Oct 31 13:42:58.208582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208301 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="47db0401-e110-4eb4-ac8a-37b7c864191e" containerName="payload-processing" Oct 31 13:42:58.208582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208308 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6783d96d-5e10-4760-9dd8-a79d495ff706" containerName="revoke-keys" Oct 31 13:42:58.208582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.208314 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d13e1c-c11e-4d70-8c1b-01c2679a2f27" containerName="istio-proxy" Oct 31 13:42:58.214179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.214154 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.234901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.231649 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck"] Oct 31 13:42:58.238425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.238388 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"openshift-service-ca.crt\"" Oct 31 13:42:58.238598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.238565 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"default-dockercfg-w85bx\"" Oct 31 13:42:58.238748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.238729 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"sub-test-model-1e72f6-kserve-self-signed-certs\"" Oct 31 13:42:58.238856 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.238828 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1e72f6\"/\"kube-root-ca.crt\"" Oct 31 13:42:58.357857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.357817 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-model-cache\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.358083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.357871 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd55587-a53c-4c0e-8642-15260308824a-tls-certs\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.358083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.358008 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-dshm\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.358232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.358132 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/2dd55587-a53c-4c0e-8642-15260308824a-kube-api-access-nn4zn\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.358232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.358218 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-tmp-dir\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.358350 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.358324 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-home\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.470651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.468881 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-home\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.470651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.468946 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-model-cache\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.470651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.468988 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd55587-a53c-4c0e-8642-15260308824a-tls-certs\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.470651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.469069 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-dshm\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.470651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.469161 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/2dd55587-a53c-4c0e-8642-15260308824a-kube-api-access-nn4zn\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.470651 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.469211 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-tmp-dir\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.499655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.496901 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-model-cache\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.499655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.498322 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-home\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.524398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.524356 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-tmp-dir\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.552723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.552681 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-dshm\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.555337 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.555287 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd55587-a53c-4c0e-8642-15260308824a-tls-certs\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.568830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.568486 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/2dd55587-a53c-4c0e-8642-15260308824a-kube-api-access-nn4zn\") pod \"sub-test-model-1e72f6-kserve-5785ccf867-nvnck\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:58.837003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:58.836896 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:42:59.155745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:59.155719 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck"] Oct 31 13:42:59.156784 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:42:59.156755 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dd55587_a53c_4c0e_8642_15260308824a.slice/crio-749be560d495c05830bab914677c8f25b063e883e31a1639ed5542415ea756b3 WatchSource:0}: Error finding container 749be560d495c05830bab914677c8f25b063e883e31a1639ed5542415ea756b3: Status 404 returned error can't find the container with id 749be560d495c05830bab914677c8f25b063e883e31a1639ed5542415ea756b3 Oct 31 13:42:59.404849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:59.404740 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" event={"ID":"2dd55587-a53c-4c0e-8642-15260308824a","Type":"ContainerStarted","Data":"eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b"} Oct 31 13:42:59.404849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:59.404801 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" event={"ID":"2dd55587-a53c-4c0e-8642-15260308824a","Type":"ContainerStarted","Data":"749be560d495c05830bab914677c8f25b063e883e31a1639ed5542415ea756b3"} Oct 31 13:42:59.437416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:59.437354 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" podStartSLOduration=1.437334031 podStartE2EDuration="1.437334031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:59.430621458 +0000 UTC m=+4273.608339445" watchObservedRunningTime="2026-10-31 13:42:59.437334031 +0000 UTC m=+4273.615052018" Oct 31 13:42:59.714335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:59.714216 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" podUID="2348a7b1-ee73-4386-8f9d-060c1f23aed6" containerName="payload-processing" containerID="cri-o://5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9" gracePeriod=25 Oct 31 13:42:59.838332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:42:59.838298 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:43:00.040488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.040459 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:43:00.104987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.104950 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2348a7b1-ee73-4386-8f9d-060c1f23aed6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2348a7b1-ee73-4386-8f9d-060c1f23aed6-plugins-config-volume\") pod \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " Oct 31 13:43:00.105183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.105056 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2348a7b1-ee73-4386-8f9d-060c1f23aed6-kube-api-access-4slhg\" (UniqueName: \"kubernetes.io/projected/2348a7b1-ee73-4386-8f9d-060c1f23aed6-kube-api-access-4slhg\") pod \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\" (UID: \"2348a7b1-ee73-4386-8f9d-060c1f23aed6\") " Oct 31 13:43:00.105392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.105358 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2348a7b1-ee73-4386-8f9d-060c1f23aed6-plugins-config-volume" pod "2348a7b1-ee73-4386-8f9d-060c1f23aed6" (UID: "2348a7b1-ee73-4386-8f9d-060c1f23aed6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:00.105808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.105786 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2348a7b1-ee73-4386-8f9d-060c1f23aed6-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:00.107301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.107276 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2348a7b1-ee73-4386-8f9d-060c1f23aed6-kube-api-access-4slhg" pod "2348a7b1-ee73-4386-8f9d-060c1f23aed6" (UID: "2348a7b1-ee73-4386-8f9d-060c1f23aed6"). InnerVolumeSpecName "kube-api-access-4slhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:00.206713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.206672 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4slhg\" (UniqueName: \"kubernetes.io/projected/2348a7b1-ee73-4386-8f9d-060c1f23aed6-kube-api-access-4slhg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:00.326481 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.326402 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf"] Oct 31 13:43:00.327050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.327035 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2348a7b1-ee73-4386-8f9d-060c1f23aed6" containerName="payload-processing" Oct 31 13:43:00.327119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.327051 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2348a7b1-ee73-4386-8f9d-060c1f23aed6" containerName="payload-processing" Oct 31 13:43:00.327158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.327141 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2348a7b1-ee73-4386-8f9d-060c1f23aed6" containerName="payload-processing" Oct 31 13:43:00.330764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.330744 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.338930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.338901 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:43:00.340425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.340229 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:43:00.340425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.340272 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:43:00.340599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.340430 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:43:00.344041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.344010 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf"] Oct 31 13:43:00.409825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.409790 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6pgg\" (UniqueName: \"kubernetes.io/projected/b771d36c-cd7c-4421-858a-73df3e933269-kube-api-access-n6pgg\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.410024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.409936 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.410024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.410005 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.412677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.412650 5074 generic.go:356] "Generic (PLEG): container finished" podID="2348a7b1-ee73-4386-8f9d-060c1f23aed6" containerID="5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9" exitCode=0 Oct 31 13:43:00.412826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.412697 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" Oct 31 13:43:00.412826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.412738 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" event={"ID":"2348a7b1-ee73-4386-8f9d-060c1f23aed6","Type":"ContainerDied","Data":"5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9"} Oct 31 13:43:00.412826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.412789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2" event={"ID":"2348a7b1-ee73-4386-8f9d-060c1f23aed6","Type":"ContainerDied","Data":"42b512b024dc58b721fc723fdf972d4707c045d4e9b745c4e7ddefdfcc26bf4d"} Oct 31 13:43:00.412826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.412820 5074 scope.go:122] "RemoveContainer" containerID="5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9" Oct 31 13:43:00.434511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.434475 5074 scope.go:122] "RemoveContainer" containerID="5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9" Oct 31 13:43:00.435111 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:00.435069 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9\": container with ID starting with 5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9 not found: ID does not exist" containerID="5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9" Oct 31 13:43:00.435216 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.435123 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9"} err="failed to get container status \"5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9\": rpc error: code = NotFound desc = could not find container \"5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9\": container with ID starting with 5184503436f8e992f7af08074768ff39266d923edef6499fa18d04c9b95b87b9 not found: ID does not exist" Oct 31 13:43:00.450539 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.450503 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2"] Oct 31 13:43:00.456202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.456174 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5dc570-545cdc76c-jmbz2"] Oct 31 13:43:00.511093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.511052 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.511317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.511294 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6pgg\" (UniqueName: \"kubernetes.io/projected/b771d36c-cd7c-4421-858a-73df3e933269-kube-api-access-n6pgg\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.511483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.511459 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.512138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.512112 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.513595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.513569 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.527105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.527082 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6pgg\" (UniqueName: \"kubernetes.io/projected/b771d36c-cd7c-4421-858a-73df3e933269-kube-api-access-n6pgg\") pod \"maas-api-revoke-sub-816023d3-bqtkf\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.647327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.647241 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:00.947745 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:00.947718 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf"] Oct 31 13:43:00.948897 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:43:00.948870 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb771d36c_cd7c_4421_858a_73df3e933269.slice/crio-f74081ef45e14229754a220007e4aeb646134e8e305576109ee9b760d66ab48c WatchSource:0}: Error finding container f74081ef45e14229754a220007e4aeb646134e8e305576109ee9b760d66ab48c: Status 404 returned error can't find the container with id f74081ef45e14229754a220007e4aeb646134e8e305576109ee9b760d66ab48c Oct 31 13:43:01.423035 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:01.422995 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" event={"ID":"b771d36c-cd7c-4421-858a-73df3e933269","Type":"ContainerStarted","Data":"4fdc0272a3684cb2a7045222a6ebdb5b5a3e2074bee6622c21b65fc7386d0ee7"} Oct 31 13:43:01.423535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:01.423061 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" event={"ID":"b771d36c-cd7c-4421-858a-73df3e933269","Type":"ContainerStarted","Data":"f74081ef45e14229754a220007e4aeb646134e8e305576109ee9b760d66ab48c"} Oct 31 13:43:01.450421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:01.450369 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" podStartSLOduration=1.450354816 podStartE2EDuration="1.450354816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:43:01.4502323 +0000 UTC m=+4275.627950275" watchObservedRunningTime="2026-10-31 13:43:01.450354816 +0000 UTC m=+4275.628072800" Oct 31 13:43:01.940244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:01.940202 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2348a7b1-ee73-4386-8f9d-060c1f23aed6" path="/var/lib/kubelet/pods/2348a7b1-ee73-4386-8f9d-060c1f23aed6/volumes" Oct 31 13:43:02.434643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:02.434595 5074 generic.go:356] "Generic (PLEG): container finished" podID="b771d36c-cd7c-4421-858a-73df3e933269" containerID="4fdc0272a3684cb2a7045222a6ebdb5b5a3e2074bee6622c21b65fc7386d0ee7" exitCode=0 Oct 31 13:43:02.435072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:02.434680 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" event={"ID":"b771d36c-cd7c-4421-858a-73df3e933269","Type":"ContainerDied","Data":"4fdc0272a3684cb2a7045222a6ebdb5b5a3e2074bee6622c21b65fc7386d0ee7"} Oct 31 13:43:03.636510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.636479 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:03.668686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.665261 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-xks-ca\") pod \"b771d36c-cd7c-4421-858a-73df3e933269\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " Oct 31 13:43:03.668686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.665421 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b771d36c-cd7c-4421-858a-73df3e933269-kube-api-access-n6pgg\" (UniqueName: \"kubernetes.io/projected/b771d36c-cd7c-4421-858a-73df3e933269-kube-api-access-n6pgg\") pod \"b771d36c-cd7c-4421-858a-73df3e933269\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " Oct 31 13:43:03.668686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.665562 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-openshift-ca\") pod \"b771d36c-cd7c-4421-858a-73df3e933269\" (UID: \"b771d36c-cd7c-4421-858a-73df3e933269\") " Oct 31 13:43:03.675343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.669687 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-openshift-ca" pod "b771d36c-cd7c-4421-858a-73df3e933269" (UID: "b771d36c-cd7c-4421-858a-73df3e933269"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:03.678568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.678534 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-xks-ca" pod "b771d36c-cd7c-4421-858a-73df3e933269" (UID: "b771d36c-cd7c-4421-858a-73df3e933269"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:03.679207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.679177 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b771d36c-cd7c-4421-858a-73df3e933269-kube-api-access-n6pgg" pod "b771d36c-cd7c-4421-858a-73df3e933269" (UID: "b771d36c-cd7c-4421-858a-73df3e933269"). InnerVolumeSpecName "kube-api-access-n6pgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:03.767450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.767410 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6pgg\" (UniqueName: \"kubernetes.io/projected/b771d36c-cd7c-4421-858a-73df3e933269-kube-api-access-n6pgg\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:03.767450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.767446 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:03.767450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:03.767457 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b771d36c-cd7c-4421-858a-73df3e933269-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:04.451185 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:04.451152 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" Oct 31 13:43:04.451375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:04.451176 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf" event={"ID":"b771d36c-cd7c-4421-858a-73df3e933269","Type":"ContainerDied","Data":"f74081ef45e14229754a220007e4aeb646134e8e305576109ee9b760d66ab48c"} Oct 31 13:43:04.451375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:04.451231 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f74081ef45e14229754a220007e4aeb646134e8e305576109ee9b760d66ab48c" Oct 31 13:43:05.433004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.432961 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6"] Oct 31 13:43:05.433567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.433552 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b771d36c-cd7c-4421-858a-73df3e933269" containerName="revoke-keys" Oct 31 13:43:05.433616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.433568 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b771d36c-cd7c-4421-858a-73df3e933269" containerName="revoke-keys" Oct 31 13:43:05.433721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.433711 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b771d36c-cd7c-4421-858a-73df3e933269" containerName="revoke-keys" Oct 31 13:43:05.437502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.437485 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.441399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.441355 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:43:05.442724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.442706 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:43:05.451101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.451075 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6"] Oct 31 13:43:05.463741 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.463710 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:43:05.463896 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.463789 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:43:05.545352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.545302 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.545536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.545370 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8h7\" (UniqueName: \"kubernetes.io/projected/9686267f-a2ad-4931-b2c0-eeb5e712540a-kube-api-access-wg8h7\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.545720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.545690 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.647260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.647213 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.647260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.647266 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.647531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.647295 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8h7\" (UniqueName: \"kubernetes.io/projected/9686267f-a2ad-4931-b2c0-eeb5e712540a-kube-api-access-wg8h7\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.648107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.648078 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.649786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.649758 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.663199 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.663172 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8h7\" (UniqueName: \"kubernetes.io/projected/9686267f-a2ad-4931-b2c0-eeb5e712540a-kube-api-access-wg8h7\") pod \"maas-api-revoke-sub-bb95dc00-tq5f6\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:05.752728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:05.752689 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:06.052541 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:43:06.052505 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9686267f_a2ad_4931_b2c0_eeb5e712540a.slice/crio-c9fe0b47703a086589afe01737bf7d9241c0de1c36d17134e0729a5cf5aa9db2 WatchSource:0}: Error finding container c9fe0b47703a086589afe01737bf7d9241c0de1c36d17134e0729a5cf5aa9db2: Status 404 returned error can't find the container with id c9fe0b47703a086589afe01737bf7d9241c0de1c36d17134e0729a5cf5aa9db2 Oct 31 13:43:06.052692 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:06.052605 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6"] Oct 31 13:43:06.467296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:06.467255 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" event={"ID":"9686267f-a2ad-4931-b2c0-eeb5e712540a","Type":"ContainerStarted","Data":"c9fe0b47703a086589afe01737bf7d9241c0de1c36d17134e0729a5cf5aa9db2"} Oct 31 13:43:07.479147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:07.479106 5074 generic.go:356] "Generic (PLEG): container finished" podID="9686267f-a2ad-4931-b2c0-eeb5e712540a" containerID="00d6f0e2608638c045df03c16559b6be3fe447f620bc355b14de9e6f4d974ddb" exitCode=0 Oct 31 13:43:07.479696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:07.479197 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" event={"ID":"9686267f-a2ad-4931-b2c0-eeb5e712540a","Type":"ContainerDied","Data":"00d6f0e2608638c045df03c16559b6be3fe447f620bc355b14de9e6f4d974ddb"} Oct 31 13:43:07.646827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:07.646790 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" containerID="cri-o://75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194" gracePeriod=45 Oct 31 13:43:07.647026 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:07.646857 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" containerID="cri-o://958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3" gracePeriod=45 Oct 31 13:43:07.663224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:07.663190 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" containerID="cri-o://953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12" gracePeriod=45 Oct 31 13:43:07.862980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:07.862929 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": dial tcp 10.132.0.103:8000: connect: connection refused" Oct 31 13:43:08.015255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.015219 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:43:08.035927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.034849 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 13:43:08.046265 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.046231 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 13:43:08.096728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.096686 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-model-cache\") pod \"4c4dc8be-7585-47cc-a538-cd3d332eee43\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " Oct 31 13:43:08.096920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.096845 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4dc8be-7585-47cc-a538-cd3d332eee43-kube-api-access-2h4qf\" (UniqueName: \"kubernetes.io/projected/4c4dc8be-7585-47cc-a538-cd3d332eee43-kube-api-access-2h4qf\") pod \"4c4dc8be-7585-47cc-a538-cd3d332eee43\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " Oct 31 13:43:08.096920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.096908 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4dc8be-7585-47cc-a538-cd3d332eee43-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4dc8be-7585-47cc-a538-cd3d332eee43-tls-certs\") pod \"4c4dc8be-7585-47cc-a538-cd3d332eee43\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " Oct 31 13:43:08.097049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.096945 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-home\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-home\") pod \"4c4dc8be-7585-47cc-a538-cd3d332eee43\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " Oct 31 13:43:08.097049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.096976 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-tmp-dir\") pod \"4c4dc8be-7585-47cc-a538-cd3d332eee43\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " Oct 31 13:43:08.097161 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.097150 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-dshm\") pod \"4c4dc8be-7585-47cc-a538-cd3d332eee43\" (UID: \"4c4dc8be-7585-47cc-a538-cd3d332eee43\") " Oct 31 13:43:08.106615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.099491 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-tmp-dir" pod "4c4dc8be-7585-47cc-a538-cd3d332eee43" (UID: "4c4dc8be-7585-47cc-a538-cd3d332eee43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.106615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.100033 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-home" pod "4c4dc8be-7585-47cc-a538-cd3d332eee43" (UID: "4c4dc8be-7585-47cc-a538-cd3d332eee43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.106615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.100090 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-model-cache" pod "4c4dc8be-7585-47cc-a538-cd3d332eee43" (UID: "4c4dc8be-7585-47cc-a538-cd3d332eee43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.110231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.110174 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4dc8be-7585-47cc-a538-cd3d332eee43-kube-api-access-2h4qf" pod "4c4dc8be-7585-47cc-a538-cd3d332eee43" (UID: "4c4dc8be-7585-47cc-a538-cd3d332eee43"). InnerVolumeSpecName "kube-api-access-2h4qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:08.111757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.111612 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-dshm" pod "4c4dc8be-7585-47cc-a538-cd3d332eee43" (UID: "4c4dc8be-7585-47cc-a538-cd3d332eee43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.111757 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.111744 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4dc8be-7585-47cc-a538-cd3d332eee43-tls-certs" pod "4c4dc8be-7585-47cc-a538-cd3d332eee43" (UID: "4c4dc8be-7585-47cc-a538-cd3d332eee43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:08.198753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.198720 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68176c7b-d2f0-425f-94e0-aadb0d173a37-tls-certs\" (UniqueName: \"kubernetes.io/secret/68176c7b-d2f0-425f-94e0-aadb0d173a37-tls-certs\") pod \"68176c7b-d2f0-425f-94e0-aadb0d173a37\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " Oct 31 13:43:08.199113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.199088 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-home\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-home\") pod \"68176c7b-d2f0-425f-94e0-aadb0d173a37\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " Oct 31 13:43:08.200698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.200673 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d85f7d-a25a-48a3-81c4-08a485ba169d-kube-api-access-d9b6m\" (UniqueName: \"kubernetes.io/projected/88d85f7d-a25a-48a3-81c4-08a485ba169d-kube-api-access-d9b6m\") pod \"88d85f7d-a25a-48a3-81c4-08a485ba169d\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " Oct 31 13:43:08.202159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.202129 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-home" pod "68176c7b-d2f0-425f-94e0-aadb0d173a37" (UID: "68176c7b-d2f0-425f-94e0-aadb0d173a37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.202583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.202321 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d85f7d-a25a-48a3-81c4-08a485ba169d-tls-certs\" (UniqueName: \"kubernetes.io/secret/88d85f7d-a25a-48a3-81c4-08a485ba169d-tls-certs\") pod \"88d85f7d-a25a-48a3-81c4-08a485ba169d\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " Oct 31 13:43:08.202815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.202790 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-model-cache\") pod \"88d85f7d-a25a-48a3-81c4-08a485ba169d\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " Oct 31 13:43:08.203354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.203335 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-tmp-dir\") pod \"68176c7b-d2f0-425f-94e0-aadb0d173a37\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " Oct 31 13:43:08.203570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.203266 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-model-cache" pod "88d85f7d-a25a-48a3-81c4-08a485ba169d" (UID: "88d85f7d-a25a-48a3-81c4-08a485ba169d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.203570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.203555 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-tmp-dir" pod "68176c7b-d2f0-425f-94e0-aadb0d173a37" (UID: "68176c7b-d2f0-425f-94e0-aadb0d173a37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.203776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.203755 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d85f7d-a25a-48a3-81c4-08a485ba169d-kube-api-access-d9b6m" pod "88d85f7d-a25a-48a3-81c4-08a485ba169d" (UID: "88d85f7d-a25a-48a3-81c4-08a485ba169d"). InnerVolumeSpecName "kube-api-access-d9b6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:08.203902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.203883 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-dshm\") pod \"88d85f7d-a25a-48a3-81c4-08a485ba169d\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " Oct 31 13:43:08.204068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.204047 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-model-cache\") pod \"68176c7b-d2f0-425f-94e0-aadb0d173a37\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " Oct 31 13:43:08.204440 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.204421 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-dshm\") pod \"68176c7b-d2f0-425f-94e0-aadb0d173a37\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " Oct 31 13:43:08.205183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.205159 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-tmp-dir\") pod \"88d85f7d-a25a-48a3-81c4-08a485ba169d\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " Oct 31 13:43:08.205349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.205331 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68176c7b-d2f0-425f-94e0-aadb0d173a37-kube-api-access-fdfr5\" (UniqueName: \"kubernetes.io/projected/68176c7b-d2f0-425f-94e0-aadb0d173a37-kube-api-access-fdfr5\") pod \"68176c7b-d2f0-425f-94e0-aadb0d173a37\" (UID: \"68176c7b-d2f0-425f-94e0-aadb0d173a37\") " Oct 31 13:43:08.205939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.204366 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-model-cache" pod "68176c7b-d2f0-425f-94e0-aadb0d173a37" (UID: "68176c7b-d2f0-425f-94e0-aadb0d173a37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.205939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.205905 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-home\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-home\") pod \"88d85f7d-a25a-48a3-81c4-08a485ba169d\" (UID: \"88d85f7d-a25a-48a3-81c4-08a485ba169d\") " Oct 31 13:43:08.205939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.205796 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-tmp-dir" pod "88d85f7d-a25a-48a3-81c4-08a485ba169d" (UID: "88d85f7d-a25a-48a3-81c4-08a485ba169d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206261 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-home" pod "88d85f7d-a25a-48a3-81c4-08a485ba169d" (UID: "88d85f7d-a25a-48a3-81c4-08a485ba169d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206595 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9b6m\" (UniqueName: \"kubernetes.io/projected/88d85f7d-a25a-48a3-81c4-08a485ba169d-kube-api-access-d9b6m\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206618 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206666 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206681 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206695 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206709 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4qf\" (UniqueName: \"kubernetes.io/projected/4c4dc8be-7585-47cc-a538-cd3d332eee43-kube-api-access-2h4qf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206723 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4dc8be-7585-47cc-a538-cd3d332eee43-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206739 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206754 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206767 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206779 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206793 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4dc8be-7585-47cc-a538-cd3d332eee43-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.206826 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.206805 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.215301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.215209 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-dshm" pod "88d85f7d-a25a-48a3-81c4-08a485ba169d" (UID: "88d85f7d-a25a-48a3-81c4-08a485ba169d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.215301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.215275 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d85f7d-a25a-48a3-81c4-08a485ba169d-tls-certs" pod "88d85f7d-a25a-48a3-81c4-08a485ba169d" (UID: "88d85f7d-a25a-48a3-81c4-08a485ba169d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:08.215301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.215272 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68176c7b-d2f0-425f-94e0-aadb0d173a37-tls-certs" pod "68176c7b-d2f0-425f-94e0-aadb0d173a37" (UID: "68176c7b-d2f0-425f-94e0-aadb0d173a37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:08.215880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.215852 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68176c7b-d2f0-425f-94e0-aadb0d173a37-kube-api-access-fdfr5" pod "68176c7b-d2f0-425f-94e0-aadb0d173a37" (UID: "68176c7b-d2f0-425f-94e0-aadb0d173a37"). InnerVolumeSpecName "kube-api-access-fdfr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:08.216599 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.216574 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-dshm" pod "68176c7b-d2f0-425f-94e0-aadb0d173a37" (UID: "68176c7b-d2f0-425f-94e0-aadb0d173a37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:08.311301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.311196 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d85f7d-a25a-48a3-81c4-08a485ba169d-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.311301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.311231 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d85f7d-a25a-48a3-81c4-08a485ba169d-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.311301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.311246 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68176c7b-d2f0-425f-94e0-aadb0d173a37-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.311301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.311259 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdfr5\" (UniqueName: \"kubernetes.io/projected/68176c7b-d2f0-425f-94e0-aadb0d173a37-kube-api-access-fdfr5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.311301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.311272 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68176c7b-d2f0-425f-94e0-aadb0d173a37-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.489326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.489292 5074 generic.go:356] "Generic (PLEG): container finished" podID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerID="75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194" exitCode=0 Oct 31 13:43:08.489839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.489349 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" Oct 31 13:43:08.489839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.489379 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" event={"ID":"4c4dc8be-7585-47cc-a538-cd3d332eee43","Type":"ContainerDied","Data":"75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194"} Oct 31 13:43:08.489839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.489432 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb" event={"ID":"4c4dc8be-7585-47cc-a538-cd3d332eee43","Type":"ContainerDied","Data":"c2b9da674d4b44a98c9c1c68f0218b1fa751f22b5de5c7f125f4f4e521c5a06c"} Oct 31 13:43:08.489839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.489455 5074 scope.go:122] "RemoveContainer" containerID="75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194" Oct 31 13:43:08.491316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.491248 5074 generic.go:356] "Generic (PLEG): container finished" podID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerID="953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12" exitCode=0 Oct 31 13:43:08.491316 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.491294 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" Oct 31 13:43:08.491515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.491385 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" event={"ID":"88d85f7d-a25a-48a3-81c4-08a485ba169d","Type":"ContainerDied","Data":"953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12"} Oct 31 13:43:08.492031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.491418 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg" event={"ID":"88d85f7d-a25a-48a3-81c4-08a485ba169d","Type":"ContainerDied","Data":"6b0a4332b4cc6fd04bf8c9af0c8a8722972cd98f90302fe58176193fd2f59e45"} Oct 31 13:43:08.494746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.494721 5074 generic.go:356] "Generic (PLEG): container finished" podID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerID="958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3" exitCode=0 Oct 31 13:43:08.494970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.494953 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" Oct 31 13:43:08.495863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.495737 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" event={"ID":"68176c7b-d2f0-425f-94e0-aadb0d173a37","Type":"ContainerDied","Data":"958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3"} Oct 31 13:43:08.495863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.495789 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7" event={"ID":"68176c7b-d2f0-425f-94e0-aadb0d173a37","Type":"ContainerDied","Data":"3b2cbe8007147d461d111e816c8d68eee45f384ae5179c022ba3066f2b3f1c89"} Oct 31 13:43:08.507139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.507091 5074 scope.go:122] "RemoveContainer" containerID="75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194" Oct 31 13:43:08.507455 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:08.507429 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194\": container with ID starting with 75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194 not found: ID does not exist" containerID="75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194" Oct 31 13:43:08.507541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.507466 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194"} err="failed to get container status \"75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194\": rpc error: code = NotFound desc = could not find container \"75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194\": container with ID starting with 75038d19cecaa8e41a95f5b635b9142afe556faf2898b893c668a23741dbe194 not found: ID does not exist" Oct 31 13:43:08.507541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.507490 5074 scope.go:122] "RemoveContainer" containerID="953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12" Oct 31 13:43:08.524568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.524541 5074 scope.go:122] "RemoveContainer" containerID="953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12" Oct 31 13:43:08.525138 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:08.525110 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12\": container with ID starting with 953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12 not found: ID does not exist" containerID="953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12" Oct 31 13:43:08.525222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.525149 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12"} err="failed to get container status \"953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12\": rpc error: code = NotFound desc = could not find container \"953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12\": container with ID starting with 953ff76c0499a07bd853fd0c903bdbb6ace84b42d2790e18129b828d7339bc12 not found: ID does not exist" Oct 31 13:43:08.525222 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.525172 5074 scope.go:122] "RemoveContainer" containerID="958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3" Oct 31 13:43:08.534015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.533984 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg"] Oct 31 13:43:08.538892 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.538439 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/premium-w1-622e03-kserve-557cbcbd4f-fc6mg"] Oct 31 13:43:08.539179 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.539011 5074 scope.go:122] "RemoveContainer" containerID="958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3" Oct 31 13:43:08.539446 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:08.539423 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3\": container with ID starting with 958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3 not found: ID does not exist" containerID="958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3" Oct 31 13:43:08.539524 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.539455 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3"} err="failed to get container status \"958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3\": rpc error: code = NotFound desc = could not find container \"958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3\": container with ID starting with 958884db7da3ee264baedb2fa134d894eefce24222283a96a2235cb1a890a7c3 not found: ID does not exist" Oct 31 13:43:08.549132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.549104 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb"] Oct 31 13:43:08.553592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.553561 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-2-w1-622e03-kserve-c69d8bf5-skmxb"] Oct 31 13:43:08.569220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.569138 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7"] Oct 31 13:43:08.576799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.574963 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-622e03/distinct-w1-622e03-kserve-5bf4f8f7f6-g4md7"] Oct 31 13:43:08.661198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.661164 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:08.818791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.818744 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9686267f-a2ad-4931-b2c0-eeb5e712540a-kube-api-access-wg8h7\" (UniqueName: \"kubernetes.io/projected/9686267f-a2ad-4931-b2c0-eeb5e712540a-kube-api-access-wg8h7\") pod \"9686267f-a2ad-4931-b2c0-eeb5e712540a\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " Oct 31 13:43:08.818993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.818869 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-openshift-ca\") pod \"9686267f-a2ad-4931-b2c0-eeb5e712540a\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " Oct 31 13:43:08.818993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.818909 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-xks-ca\") pod \"9686267f-a2ad-4931-b2c0-eeb5e712540a\" (UID: \"9686267f-a2ad-4931-b2c0-eeb5e712540a\") " Oct 31 13:43:08.819351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.819288 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-openshift-ca" pod "9686267f-a2ad-4931-b2c0-eeb5e712540a" (UID: "9686267f-a2ad-4931-b2c0-eeb5e712540a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:08.819578 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.819559 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.821042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.821015 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-xks-ca" pod "9686267f-a2ad-4931-b2c0-eeb5e712540a" (UID: "9686267f-a2ad-4931-b2c0-eeb5e712540a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:08.821159 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.821036 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9686267f-a2ad-4931-b2c0-eeb5e712540a-kube-api-access-wg8h7" pod "9686267f-a2ad-4931-b2c0-eeb5e712540a" (UID: "9686267f-a2ad-4931-b2c0-eeb5e712540a"). InnerVolumeSpecName "kube-api-access-wg8h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:08.838249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.838214 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:43:08.851873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.851843 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:43:08.920865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.920824 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg8h7\" (UniqueName: \"kubernetes.io/projected/9686267f-a2ad-4931-b2c0-eeb5e712540a-kube-api-access-wg8h7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:08.920865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:08.920859 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9686267f-a2ad-4931-b2c0-eeb5e712540a-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:09.503917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:09.503882 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" Oct 31 13:43:09.504398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:09.503884 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6" event={"ID":"9686267f-a2ad-4931-b2c0-eeb5e712540a","Type":"ContainerDied","Data":"c9fe0b47703a086589afe01737bf7d9241c0de1c36d17134e0729a5cf5aa9db2"} Oct 31 13:43:09.504398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:09.504059 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9fe0b47703a086589afe01737bf7d9241c0de1c36d17134e0729a5cf5aa9db2" Oct 31 13:43:09.523345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:09.523314 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:43:09.939121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:09.939032 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" path="/var/lib/kubelet/pods/4c4dc8be-7585-47cc-a538-cd3d332eee43/volumes" Oct 31 13:43:09.939580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:09.939556 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" path="/var/lib/kubelet/pods/68176c7b-d2f0-425f-94e0-aadb0d173a37/volumes" Oct 31 13:43:09.940092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:09.940070 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" path="/var/lib/kubelet/pods/88d85f7d-a25a-48a3-81c4-08a485ba169d/volumes" Oct 31 13:43:13.158568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:13.158532 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924"] Oct 31 13:43:13.174603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:13.174570 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e34628e-98924"] Oct 31 13:43:13.946797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:13.946762 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ba542c-566c-4c81-970f-969d0a6e7153" path="/var/lib/kubelet/pods/07ba542c-566c-4c81-970f-969d0a6e7153/volumes" Oct 31 13:43:19.327495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:19.327453 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz"] Oct 31 13:43:22.776600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:22.776558 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x"] Oct 31 13:43:25.988717 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:25.988677 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:43:27.778006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:27.777960 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" podUID="6eb5dffd-8093-45ec-aada-8e485a60b178" containerName="payload-pre-processing" containerID="cri-o://1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9" gracePeriod=25 Oct 31 13:43:28.734365 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:28.734324 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8"] Oct 31 13:43:28.745619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:28.745575 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9"] Oct 31 13:43:32.218328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:32.218287 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" podUID="6eb5dffd-8093-45ec-aada-8e485a60b178" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:33.735694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:33.735604 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" podUID="c8d9c999-6423-4ed8-9b67-47eda9032560" containerName="payload-processing" containerID="cri-o://d227eaac233f92c94637af79b642070761909dcedc5f6276c5118e208a0eda81" gracePeriod=25 Oct 31 13:43:33.746217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:33.746184 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" podUID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" containerName="payload-pre-processing" containerID="cri-o://795d67acbbac53dfa8cba1d3db3b992e59172b56a87f2a1aa81de919a80199a8" gracePeriod=25 Oct 31 13:43:34.364670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.360941 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" podUID="26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" containerName="main" containerID="cri-o://795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca" gracePeriod=45 Oct 31 13:43:34.673368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.673332 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:43:34.741581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.741546 5074 generic.go:356] "Generic (PLEG): container finished" podID="26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" containerID="795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca" exitCode=0 Oct 31 13:43:34.742052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.741614 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" Oct 31 13:43:34.742052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.741666 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" event={"ID":"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba","Type":"ContainerDied","Data":"795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca"} Oct 31 13:43:34.742052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.741718 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz" event={"ID":"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba","Type":"ContainerDied","Data":"a28fb4a4df25c1745250d97c85cb9c0915242e5726794206a48f84e18a1bfc80"} Oct 31 13:43:34.742052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.741739 5074 scope.go:122] "RemoveContainer" containerID="795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca" Oct 31 13:43:34.753349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.753326 5074 scope.go:122] "RemoveContainer" containerID="795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca" Oct 31 13:43:34.753664 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:34.753625 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca\": container with ID starting with 795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca not found: ID does not exist" containerID="795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca" Oct 31 13:43:34.753791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.753670 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca"} err="failed to get container status \"795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca\": rpc error: code = NotFound desc = could not find container \"795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca\": container with ID starting with 795795518c28a26ab42aff7d04acf01d75c1df73214d6cf3d67d465a84a65bca not found: ID does not exist" Oct 31 13:43:34.780409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780376 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-model-cache\") pod \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " Oct 31 13:43:34.780617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780458 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-dshm\") pod \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " Oct 31 13:43:34.780617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780496 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tls-certs\") pod \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " Oct 31 13:43:34.780617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780539 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-kube-api-access-gj9t4\" (UniqueName: \"kubernetes.io/projected/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-kube-api-access-gj9t4\") pod \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " Oct 31 13:43:34.780617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780571 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-home\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-home\") pod \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " Oct 31 13:43:34.780905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780715 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-model-cache" pod "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" (UID: "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:34.780905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780761 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tmp-dir\") pod \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\" (UID: \"26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba\") " Oct 31 13:43:34.781005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.780935 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-home" pod "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" (UID: "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:34.781224 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.781199 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tmp-dir" pod "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" (UID: "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:34.781413 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.781396 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:34.781463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.781420 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:34.781463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.781436 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:34.782927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.782898 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-kube-api-access-gj9t4" pod "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" (UID: "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba"). InnerVolumeSpecName "kube-api-access-gj9t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:34.783117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.783099 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-dshm" pod "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" (UID: "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:34.783421 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.783402 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tls-certs" pod "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" (UID: "26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:43:34.892780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.883938 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:34.892780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.883980 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:34.892780 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:34.883994 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9t4\" (UniqueName: \"kubernetes.io/projected/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba-kube-api-access-gj9t4\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:35.095968 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:35.095931 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz"] Oct 31 13:43:35.112438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:35.112404 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5452fc/rate-test-model-5452fc-kserve-7957bdc47c-6vqbz"] Oct 31 13:43:35.945526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:35.945484 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" path="/var/lib/kubelet/pods/26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba/volumes" Oct 31 13:43:36.079442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:36.079406 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:43:36.988844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:36.988804 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq"] Oct 31 13:43:36.989650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:36.989595 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" containerID="cri-o://f06a7411202d801df9503d2529b194a79d71e37e7338ce0366c053052cbfb972" gracePeriod=30 Oct 31 13:43:38.203763 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.203723 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq"] Oct 31 13:43:38.204611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204586 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9686267f-a2ad-4931-b2c0-eeb5e712540a" containerName="revoke-keys" Oct 31 13:43:38.204696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204614 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9686267f-a2ad-4931-b2c0-eeb5e712540a" containerName="revoke-keys" Oct 31 13:43:38.204696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204650 5074 cpu_manager.go:414] "containerMap: removing container" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" Oct 31 13:43:38.204696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204660 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" Oct 31 13:43:38.204696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204671 5074 cpu_manager.go:414] "containerMap: removing container" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" Oct 31 13:43:38.204696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204679 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" Oct 31 13:43:38.204868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204707 5074 cpu_manager.go:414] "containerMap: removing container" podUID="26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" containerName="main" Oct 31 13:43:38.204868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204715 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" containerName="main" Oct 31 13:43:38.204868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204747 5074 cpu_manager.go:414] "containerMap: removing container" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" Oct 31 13:43:38.204868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204754 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" Oct 31 13:43:38.204995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204919 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9686267f-a2ad-4931-b2c0-eeb5e712540a" containerName="revoke-keys" Oct 31 13:43:38.204995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204935 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="68176c7b-d2f0-425f-94e0-aadb0d173a37" containerName="main" Oct 31 13:43:38.204995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204946 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4dc8be-7585-47cc-a538-cd3d332eee43" containerName="main" Oct 31 13:43:38.204995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204958 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cd6bc4-3d7c-4169-a4f8-c4fa1bf283ba" containerName="main" Oct 31 13:43:38.204995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.204972 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d85f7d-a25a-48a3-81c4-08a485ba169d" containerName="main" Oct 31 13:43:38.210329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.210307 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.234567 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.234538 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq"] Oct 31 13:43:38.293853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.293823 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6c0612-openshift-default-dockercfg-thqpk\"" Oct 31 13:43:38.302656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.302613 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c12e2a-9e24-4adf-be3c-279947b46915-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.302782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.302663 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-certs\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.302782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.302746 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c12e2a-9e24-4adf-be3c-279947b46915-istio-podinfo\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.302873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.302843 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrss\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-kube-api-access-2zrss\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.302873 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.302866 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-envoy\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.302952 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.302939 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-istio-token\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.303020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.303004 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.303078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.303034 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-credential-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.303180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.303157 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-data\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.404376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404332 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-data\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.404549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404405 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c12e2a-9e24-4adf-be3c-279947b46915-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.404549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404424 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-certs\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.404549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404453 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c12e2a-9e24-4adf-be3c-279947b46915-istio-podinfo\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.404549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404526 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrss\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-kube-api-access-2zrss\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.404878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404551 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-envoy\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.404878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404662 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-istio-token\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.405354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.404829 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-data\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.405354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.405119 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.405354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.405207 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-credential-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.405354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.405226 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-certs\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.405354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.405209 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c12e2a-9e24-4adf-be3c-279947b46915-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.405775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.405437 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.405775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.405454 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-credential-socket\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.407074 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.407052 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-envoy\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.407299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.407281 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c12e2a-9e24-4adf-be3c-279947b46915-istio-podinfo\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.434432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.434394 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-istio-token\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.434618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.434585 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrss\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-kube-api-access-2zrss\") pod \"e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.524252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.524222 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:38.827677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.827647 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq"] Oct 31 13:43:38.828787 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:43:38.828759 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c12e2a_9e24_4adf_be3c_279947b46915.slice/crio-a7be44a1653e70200c1c22e15f247c8ab01b282b89c4c7e57d82ab594dee10ef WatchSource:0}: Error finding container a7be44a1653e70200c1c22e15f247c8ab01b282b89c4c7e57d82ab594dee10ef: Status 404 returned error can't find the container with id a7be44a1653e70200c1c22e15f247c8ab01b282b89c4c7e57d82ab594dee10ef Oct 31 13:43:38.831256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.831222 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:38.831343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.831291 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:38.831343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:38.831322 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:43:39.803095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:39.803049 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" event={"ID":"83c12e2a-9e24-4adf-be3c-279947b46915","Type":"ContainerStarted","Data":"7970aae06c487e6bbeea814effc79e7e40c5683ba353c514a1248932ffa67ba8"} Oct 31 13:43:39.803537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:39.803185 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" event={"ID":"83c12e2a-9e24-4adf-be3c-279947b46915","Type":"ContainerStarted","Data":"a7be44a1653e70200c1c22e15f247c8ab01b282b89c4c7e57d82ab594dee10ef"} Oct 31 13:43:39.845808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:39.845743 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" podStartSLOduration=1.845727334 podStartE2EDuration="1.845727334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:43:39.842868212 +0000 UTC m=+4314.020586197" watchObservedRunningTime="2026-10-31 13:43:39.845727334 +0000 UTC m=+4314.023445315" Oct 31 13:43:40.525000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:40.524966 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:40.526857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:40.526830 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 13:43:40.526979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:40.526883 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 13:43:40.729752 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:40.729712 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" podUID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:40.729752 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:40.729740 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" podUID="c8d9c999-6423-4ed8-9b67-47eda9032560" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:41.525656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:41.525168 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 13:43:41.525656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:41.525231 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 13:43:42.234738 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:42.234697 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" podUID="6eb5dffd-8093-45ec-aada-8e485a60b178" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:42.525482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:42.525021 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 13:43:42.525482 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:42.525089 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 13:43:42.833621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:42.833533 5074 generic.go:356] "Generic (PLEG): container finished" podID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerID="f06a7411202d801df9503d2529b194a79d71e37e7338ce0366c053052cbfb972" exitCode=0 Oct 31 13:43:42.833621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:42.833601 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" event={"ID":"a22c34a8-e6b9-4e0a-bd18-177413888baa","Type":"ContainerDied","Data":"f06a7411202d801df9503d2529b194a79d71e37e7338ce0366c053052cbfb972"} Oct 31 13:43:43.419981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.419480 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422188 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-podinfo\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422318 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-socket\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422348 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-data\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422410 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22c34a8-e6b9-4e0a-bd18-177413888baa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22c34a8-e6b9-4e0a-bd18-177413888baa-istiod-ca-cert\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422466 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-kube-api-access-qsl2r\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-kube-api-access-qsl2r\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422503 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-socket" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422529 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-certs\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422647 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-envoy\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422681 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-token\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-token\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422750 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-credential-socket\") pod \"a22c34a8-e6b9-4e0a-bd18-177413888baa\" (UID: \"a22c34a8-e6b9-4e0a-bd18-177413888baa\") " Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422787 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22c34a8-e6b9-4e0a-bd18-177413888baa-istiod-ca-cert" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.422949 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-data" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.423073 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-certs" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.423270 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.423289 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.423304 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22c34a8-e6b9-4e0a-bd18-177413888baa-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.423544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.423318 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.424974 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.424627 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-credential-socket" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:43.426595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.426559 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-podinfo" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:43.426926 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.426894 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-token" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:43.427045 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.427022 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-kube-api-access-qsl2r" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "kube-api-access-qsl2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:43.428565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.428542 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-envoy" pod "a22c34a8-e6b9-4e0a-bd18-177413888baa" (UID: "a22c34a8-e6b9-4e0a-bd18-177413888baa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:43.649456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.649355 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.649456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.649391 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.649456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.649405 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22c34a8-e6b9-4e0a-bd18-177413888baa-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.649456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.649420 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22c34a8-e6b9-4e0a-bd18-177413888baa-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.649456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.649432 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsl2r\" (UniqueName: \"kubernetes.io/projected/a22c34a8-e6b9-4e0a-bd18-177413888baa-kube-api-access-qsl2r\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:43.651588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.650770 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 13:43:43.651588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.650824 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 13:43:43.845682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.842772 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" event={"ID":"a22c34a8-e6b9-4e0a-bd18-177413888baa","Type":"ContainerDied","Data":"8bc43761bc09cd98293ea30387b85b265614c4ea294088d2f5166047fff60bcb"} Oct 31 13:43:43.845682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.842833 5074 scope.go:122] "RemoveContainer" containerID="f06a7411202d801df9503d2529b194a79d71e37e7338ce0366c053052cbfb972" Oct 31 13:43:43.845682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.842960 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" Oct 31 13:43:43.903865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.903822 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq"] Oct 31 13:43:43.907660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.907608 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq"] Oct 31 13:43:43.939274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:43.939239 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" path="/var/lib/kubelet/pods/a22c34a8-e6b9-4e0a-bd18-177413888baa/volumes" Oct 31 13:43:44.338852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:44.338811 5074 patch_prober.go:29] interesting pod/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:43:44.339014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:44.338871 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-622e03-openshift-default-57f947795f-x9jqq" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:43:44.525112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:44.525067 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Oct 31 13:43:44.525298 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:44.525122 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Oct 31 13:43:45.530205 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:45.530171 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:45.530691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:45.530589 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:45.531391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:45.531368 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:43:48.075368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.074991 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:43:48.107703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.106789 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb5dffd-8093-45ec-aada-8e485a60b178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb5dffd-8093-45ec-aada-8e485a60b178-plugins-config-volume\") pod \"6eb5dffd-8093-45ec-aada-8e485a60b178\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " Oct 31 13:43:48.107703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.106851 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb5dffd-8093-45ec-aada-8e485a60b178-kube-api-access-nt4j5\" (UniqueName: \"kubernetes.io/projected/6eb5dffd-8093-45ec-aada-8e485a60b178-kube-api-access-nt4j5\") pod \"6eb5dffd-8093-45ec-aada-8e485a60b178\" (UID: \"6eb5dffd-8093-45ec-aada-8e485a60b178\") " Oct 31 13:43:48.108592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.108562 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb5dffd-8093-45ec-aada-8e485a60b178-plugins-config-volume" pod "6eb5dffd-8093-45ec-aada-8e485a60b178" (UID: "6eb5dffd-8093-45ec-aada-8e485a60b178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:48.111207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.111184 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb5dffd-8093-45ec-aada-8e485a60b178-kube-api-access-nt4j5" pod "6eb5dffd-8093-45ec-aada-8e485a60b178" (UID: "6eb5dffd-8093-45ec-aada-8e485a60b178"). InnerVolumeSpecName "kube-api-access-nt4j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:48.208486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.208452 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb5dffd-8093-45ec-aada-8e485a60b178-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:48.208486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.208483 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4j5\" (UniqueName: \"kubernetes.io/projected/6eb5dffd-8093-45ec-aada-8e485a60b178-kube-api-access-nt4j5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:48.896380 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.896321 5074 generic.go:356] "Generic (PLEG): container finished" podID="6eb5dffd-8093-45ec-aada-8e485a60b178" containerID="1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9" exitCode=0 Oct 31 13:43:48.896594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.896372 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" event={"ID":"6eb5dffd-8093-45ec-aada-8e485a60b178","Type":"ContainerDied","Data":"1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9"} Oct 31 13:43:48.896594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.896399 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" Oct 31 13:43:48.896594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.896439 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x" event={"ID":"6eb5dffd-8093-45ec-aada-8e485a60b178","Type":"ContainerDied","Data":"25ca39c166ec34460924fa3e0d262ebd42ef6e0ea931ebd12c8507a7464cb5b4"} Oct 31 13:43:48.896594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.896464 5074 scope.go:122] "RemoveContainer" containerID="1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9" Oct 31 13:43:48.940768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.940715 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x"] Oct 31 13:43:48.945568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.945498 5074 scope.go:122] "RemoveContainer" containerID="1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9" Oct 31 13:43:48.946033 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:48.946007 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9\": container with ID starting with 1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9 not found: ID does not exist" containerID="1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9" Oct 31 13:43:48.946180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.946039 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9"} err="failed to get container status \"1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9\": rpc error: code = NotFound desc = could not find container \"1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9\": container with ID starting with 1baad43e8f160fb2bcc0fbbaa508b115d1c25c7db1b2b05320b29ff60ae3b0a9 not found: ID does not exist" Oct 31 13:43:48.949533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:48.949503 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c2c98b-6847fcbc99-j297x"] Oct 31 13:43:49.939232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:49.939195 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb5dffd-8093-45ec-aada-8e485a60b178" path="/var/lib/kubelet/pods/6eb5dffd-8093-45ec-aada-8e485a60b178/volumes" Oct 31 13:43:50.729870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:50.729830 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" podUID="c8d9c999-6423-4ed8-9b67-47eda9032560" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:50.730151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:50.729905 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" podUID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:43:53.428538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:53.428499 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p"] Oct 31 13:43:53.429068 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:53.428713 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" containerID="cri-o://a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320" gracePeriod=30 Oct 31 13:43:53.949451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:53.949404 5074 generic.go:356] "Generic (PLEG): container finished" podID="c8d9c999-6423-4ed8-9b67-47eda9032560" containerID="d227eaac233f92c94637af79b642070761909dcedc5f6276c5118e208a0eda81" exitCode=0 Oct 31 13:43:53.951476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:53.951444 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" event={"ID":"c8d9c999-6423-4ed8-9b67-47eda9032560","Type":"ContainerDied","Data":"d227eaac233f92c94637af79b642070761909dcedc5f6276c5118e208a0eda81"} Oct 31 13:43:53.953554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:53.953522 5074 generic.go:356] "Generic (PLEG): container finished" podID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" containerID="795d67acbbac53dfa8cba1d3db3b992e59172b56a87f2a1aa81de919a80199a8" exitCode=0 Oct 31 13:43:53.953727 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:53.953653 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" event={"ID":"4f7723da-6f6e-4f6b-ada0-76fb6af99085","Type":"ContainerDied","Data":"795d67acbbac53dfa8cba1d3db3b992e59172b56a87f2a1aa81de919a80199a8"} Oct 31 13:43:54.051294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.050997 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:43:54.110208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.110145 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:43:54.114478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.114443 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d9c999-6423-4ed8-9b67-47eda9032560-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d9c999-6423-4ed8-9b67-47eda9032560-plugins-config-volume\") pod \"c8d9c999-6423-4ed8-9b67-47eda9032560\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " Oct 31 13:43:54.114677 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.114591 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d9c999-6423-4ed8-9b67-47eda9032560-kube-api-access-hf2qs\" (UniqueName: \"kubernetes.io/projected/c8d9c999-6423-4ed8-9b67-47eda9032560-kube-api-access-hf2qs\") pod \"c8d9c999-6423-4ed8-9b67-47eda9032560\" (UID: \"c8d9c999-6423-4ed8-9b67-47eda9032560\") " Oct 31 13:43:54.115135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.115108 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d9c999-6423-4ed8-9b67-47eda9032560-plugins-config-volume" pod "c8d9c999-6423-4ed8-9b67-47eda9032560" (UID: "c8d9c999-6423-4ed8-9b67-47eda9032560"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:54.115556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.115525 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d9c999-6423-4ed8-9b67-47eda9032560-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.118831 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.118794 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d9c999-6423-4ed8-9b67-47eda9032560-kube-api-access-hf2qs" pod "c8d9c999-6423-4ed8-9b67-47eda9032560" (UID: "c8d9c999-6423-4ed8-9b67-47eda9032560"). InnerVolumeSpecName "kube-api-access-hf2qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:54.216701 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.216648 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7723da-6f6e-4f6b-ada0-76fb6af99085-kube-api-access-rfqlb\" (UniqueName: \"kubernetes.io/projected/4f7723da-6f6e-4f6b-ada0-76fb6af99085-kube-api-access-rfqlb\") pod \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " Oct 31 13:43:54.216878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.216841 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7723da-6f6e-4f6b-ada0-76fb6af99085-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7723da-6f6e-4f6b-ada0-76fb6af99085-plugins-config-volume\") pod \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\" (UID: \"4f7723da-6f6e-4f6b-ada0-76fb6af99085\") " Oct 31 13:43:54.217287 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.217252 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7723da-6f6e-4f6b-ada0-76fb6af99085-plugins-config-volume" pod "4f7723da-6f6e-4f6b-ada0-76fb6af99085" (UID: "4f7723da-6f6e-4f6b-ada0-76fb6af99085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:54.217546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.217524 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7723da-6f6e-4f6b-ada0-76fb6af99085-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.217622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.217551 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf2qs\" (UniqueName: \"kubernetes.io/projected/c8d9c999-6423-4ed8-9b67-47eda9032560-kube-api-access-hf2qs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.219038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.219008 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7723da-6f6e-4f6b-ada0-76fb6af99085-kube-api-access-rfqlb" pod "4f7723da-6f6e-4f6b-ada0-76fb6af99085" (UID: "4f7723da-6f6e-4f6b-ada0-76fb6af99085"). InnerVolumeSpecName "kube-api-access-rfqlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:54.318611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.318513 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfqlb\" (UniqueName: \"kubernetes.io/projected/4f7723da-6f6e-4f6b-ada0-76fb6af99085-kube-api-access-rfqlb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:54.962868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.962828 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" Oct 31 13:43:54.963353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.962842 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8" event={"ID":"c8d9c999-6423-4ed8-9b67-47eda9032560","Type":"ContainerDied","Data":"0151ed835ea0d69ae6ca46062177b859be41a89074873c15d837d4c0556abc9a"} Oct 31 13:43:54.963353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.962998 5074 scope.go:122] "RemoveContainer" containerID="d227eaac233f92c94637af79b642070761909dcedc5f6276c5118e208a0eda81" Oct 31 13:43:54.964855 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.964836 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" Oct 31 13:43:54.965090 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.964847 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9" event={"ID":"4f7723da-6f6e-4f6b-ada0-76fb6af99085","Type":"ContainerDied","Data":"8c11d04b53cc8bcd7e3a5ac388a5746674945049f9c002e88f64afe183c5ff8c"} Oct 31 13:43:54.984348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:54.984317 5074 scope.go:122] "RemoveContainer" containerID="795d67acbbac53dfa8cba1d3db3b992e59172b56a87f2a1aa81de919a80199a8" Oct 31 13:43:55.016335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:55.016258 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8"] Oct 31 13:43:55.023323 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:55.023289 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2ef686-65bd7445c-x5ws8"] Oct 31 13:43:55.042162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:55.042127 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9"] Oct 31 13:43:55.050408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:55.050370 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2ef686-5f57755647-4zrj9"] Oct 31 13:43:55.941618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:55.941583 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" path="/var/lib/kubelet/pods/4f7723da-6f6e-4f6b-ada0-76fb6af99085/volumes" Oct 31 13:43:55.942048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:55.942029 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d9c999-6423-4ed8-9b67-47eda9032560" path="/var/lib/kubelet/pods/c8d9c999-6423-4ed8-9b67-47eda9032560/volumes" Oct 31 13:43:55.984177 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:55.984139 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:43:56.446897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:56.446867 5074 scope.go:122] "RemoveContainer" containerID="94d3da0192260b7e2d840e20647446c1109329808e2dad6481754a070a3654fb" Oct 31 13:43:56.459368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:56.459344 5074 scope.go:122] "RemoveContainer" containerID="68871357b20a5e22a78116af5bdab354677b1b2ba842e4ec11428ec22dc33d02" Oct 31 13:43:58.765588 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.765553 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:43:58.890664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.888146 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-podinfo\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.912309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.912276 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-envoy\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.912649 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.912613 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-socket\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.912857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.912840 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-data\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.912991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.912976 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-kube-api-access-dwvvt\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-kube-api-access-dwvvt\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.916650 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.913169 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-certs\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.916857 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.916832 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istiod-ca-cert\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.925655 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.916978 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-token\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-token\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.926032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.910621 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-podinfo" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:43:58.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.914266 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-certs" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.914280 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-data" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.914489 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-socket" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.917176 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istiod-ca-cert" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:43:58.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.918737 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-kube-api-access-dwvvt" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "kube-api-access-dwvvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:58.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.922088 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-envoy" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.926116 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.926002 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-credential-socket\") pod \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\" (UID: \"79df00a8-4a3c-4d95-a9e3-f3d8da921927\") " Oct 31 13:43:58.926715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.926691 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-credential-socket" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:43:58.926847 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.926721 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.926962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.926947 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.927058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.927044 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.927146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.927134 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwvvt\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-kube-api-access-dwvvt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.927225 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.927213 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.927312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.927300 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.927476 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.927460 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:58.938755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.938724 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-token" pod "79df00a8-4a3c-4d95-a9e3-f3d8da921927" (UID: "79df00a8-4a3c-4d95-a9e3-f3d8da921927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:43:58.997375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.997286 5074 generic.go:356] "Generic (PLEG): container finished" podID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerID="a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320" exitCode=0 Oct 31 13:43:58.997375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.997333 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" Oct 31 13:43:58.997375 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.997352 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" event={"ID":"79df00a8-4a3c-4d95-a9e3-f3d8da921927","Type":"ContainerDied","Data":"a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320"} Oct 31 13:43:58.997643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.997404 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p" event={"ID":"79df00a8-4a3c-4d95-a9e3-f3d8da921927","Type":"ContainerDied","Data":"3df2fb85823156fea50d6ddac6bcb29eb143c09ae538b407ee1000870856e9b1"} Oct 31 13:43:58.997643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:58.997434 5074 scope.go:122] "RemoveContainer" containerID="a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320" Oct 31 13:43:59.008772 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:59.008749 5074 scope.go:122] "RemoveContainer" containerID="a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320" Oct 31 13:43:59.009084 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:43:59.009056 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320\": container with ID starting with a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320 not found: ID does not exist" containerID="a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320" Oct 31 13:43:59.009162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:59.009095 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320"} err="failed to get container status \"a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320\": rpc error: code = NotFound desc = could not find container \"a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320\": container with ID starting with a7e9d737512c87c7face2fbf0c6091b9a69a406349b9d19aec2262a37afb9320 not found: ID does not exist" Oct 31 13:43:59.028944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:59.028916 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79df00a8-4a3c-4d95-a9e3-f3d8da921927-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.029058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:59.028947 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79df00a8-4a3c-4d95-a9e3-f3d8da921927-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:43:59.029417 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:59.029399 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p"] Oct 31 13:43:59.036581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:59.036557 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-45bcad-openshift-default-75d4b66bf7-6z28p"] Oct 31 13:43:59.939162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:43:59.939129 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" path="/var/lib/kubelet/pods/79df00a8-4a3c-4d95-a9e3-f3d8da921927/volumes" Oct 31 13:44:02.008799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:02.008765 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:44:11.937473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:11.937432 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8"] Oct 31 13:44:11.938376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:11.938333 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" containerID="cri-o://3d5d8ff86fc5aae05767dfd4564eb6c6ed29334ee0f85be1d9031636474acf6b" gracePeriod=30 Oct 31 13:44:12.836235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836197 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs"] Oct 31 13:44:12.836794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836772 5074 cpu_manager.go:414] "containerMap: removing container" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" Oct 31 13:44:12.836844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836795 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" Oct 31 13:44:12.836844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836806 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c8d9c999-6423-4ed8-9b67-47eda9032560" containerName="payload-processing" Oct 31 13:44:12.836844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836812 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d9c999-6423-4ed8-9b67-47eda9032560" containerName="payload-processing" Oct 31 13:44:12.836844 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836843 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836850 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836865 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6eb5dffd-8093-45ec-aada-8e485a60b178" containerName="payload-pre-processing" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836871 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb5dffd-8093-45ec-aada-8e485a60b178" containerName="payload-pre-processing" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836880 5074 cpu_manager.go:414] "containerMap: removing container" podUID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" containerName="payload-pre-processing" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836885 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" containerName="payload-pre-processing" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836966 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22c34a8-e6b9-4e0a-bd18-177413888baa" containerName="istio-proxy" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836977 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb5dffd-8093-45ec-aada-8e485a60b178" containerName="payload-pre-processing" Oct 31 13:44:12.836985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836982 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7723da-6f6e-4f6b-ada0-76fb6af99085" containerName="payload-pre-processing" Oct 31 13:44:12.837234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836993 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="79df00a8-4a3c-4d95-a9e3-f3d8da921927" containerName="istio-proxy" Oct 31 13:44:12.837234 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.836999 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d9c999-6423-4ed8-9b67-47eda9032560" containerName="payload-processing" Oct 31 13:44:12.842157 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.842133 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:12.846510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.846484 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:44:12.847557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.847535 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:44:12.847684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.847605 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:44:12.847684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.847620 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:44:12.859584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.859556 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs"] Oct 31 13:44:12.943656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.943596 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:12.944043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.943799 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:12.944043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:12.943900 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwf9\" (UniqueName: \"kubernetes.io/projected/f25de364-7b78-4e72-b4cb-ca3787b29fe1-kube-api-access-kxwf9\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.045154 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.045121 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.045338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.045208 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwf9\" (UniqueName: \"kubernetes.io/projected/f25de364-7b78-4e72-b4cb-ca3787b29fe1-kube-api-access-kxwf9\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.045338 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.045259 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.045952 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.045920 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.047678 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.047657 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.059063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.059040 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwf9\" (UniqueName: \"kubernetes.io/projected/f25de364-7b78-4e72-b4cb-ca3787b29fe1-kube-api-access-kxwf9\") pod \"maas-api-revoke-sub-b68bc77b-cdccs\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.157814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.157534 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:13.451916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:13.451884 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs"] Oct 31 13:44:13.453789 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:44:13.453750 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25de364_7b78_4e72_b4cb_ca3787b29fe1.slice/crio-e58c6c5db7c3a7599c622d66f475c42a8ca2391d9799d39b920c5d197e198332 WatchSource:0}: Error finding container e58c6c5db7c3a7599c622d66f475c42a8ca2391d9799d39b920c5d197e198332: Status 404 returned error can't find the container with id e58c6c5db7c3a7599c622d66f475c42a8ca2391d9799d39b920c5d197e198332 Oct 31 13:44:14.122961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:14.122859 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" event={"ID":"f25de364-7b78-4e72-b4cb-ca3787b29fe1","Type":"ContainerStarted","Data":"f7681e6327bbf8980594ab00f26ca91fab7062cb33a07f72223cbbbc93873245"} Oct 31 13:44:14.122961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:14.122927 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" event={"ID":"f25de364-7b78-4e72-b4cb-ca3787b29fe1","Type":"ContainerStarted","Data":"e58c6c5db7c3a7599c622d66f475c42a8ca2391d9799d39b920c5d197e198332"} Oct 31 13:44:14.151460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:14.151411 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" podStartSLOduration=2.151397925 podStartE2EDuration="2.151397925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:14.148439577 +0000 UTC m=+4348.326157562" watchObservedRunningTime="2026-10-31 13:44:14.151397925 +0000 UTC m=+4348.329115908" Oct 31 13:44:15.134400 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:15.134360 5074 generic.go:356] "Generic (PLEG): container finished" podID="f25de364-7b78-4e72-b4cb-ca3787b29fe1" containerID="f7681e6327bbf8980594ab00f26ca91fab7062cb33a07f72223cbbbc93873245" exitCode=0 Oct 31 13:44:15.134894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:15.134443 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" event={"ID":"f25de364-7b78-4e72-b4cb-ca3787b29fe1","Type":"ContainerDied","Data":"f7681e6327bbf8980594ab00f26ca91fab7062cb33a07f72223cbbbc93873245"} Oct 31 13:44:15.725819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:15.725387 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq"] Oct 31 13:44:15.725819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:15.725665 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" containerID="cri-o://1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1" gracePeriod=30 Oct 31 13:44:16.327886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.327859 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:16.499305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.499256 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25de364-7b78-4e72-b4cb-ca3787b29fe1-kube-api-access-kxwf9\" (UniqueName: \"kubernetes.io/projected/f25de364-7b78-4e72-b4cb-ca3787b29fe1-kube-api-access-kxwf9\") pod \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " Oct 31 13:44:16.499511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.499374 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-openshift-ca\") pod \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " Oct 31 13:44:16.499511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.499497 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-xks-ca\") pod \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\" (UID: \"f25de364-7b78-4e72-b4cb-ca3787b29fe1\") " Oct 31 13:44:16.499790 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.499762 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-openshift-ca" pod "f25de364-7b78-4e72-b4cb-ca3787b29fe1" (UID: "f25de364-7b78-4e72-b4cb-ca3787b29fe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:16.500142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.500111 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:16.501612 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.501588 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25de364-7b78-4e72-b4cb-ca3787b29fe1-kube-api-access-kxwf9" pod "f25de364-7b78-4e72-b4cb-ca3787b29fe1" (UID: "f25de364-7b78-4e72-b4cb-ca3787b29fe1"). InnerVolumeSpecName "kube-api-access-kxwf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:16.501762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.501627 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-xks-ca" pod "f25de364-7b78-4e72-b4cb-ca3787b29fe1" (UID: "f25de364-7b78-4e72-b4cb-ca3787b29fe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:16.601590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.601547 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwf9\" (UniqueName: \"kubernetes.io/projected/f25de364-7b78-4e72-b4cb-ca3787b29fe1-kube-api-access-kxwf9\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:16.601590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:16.601590 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25de364-7b78-4e72-b4cb-ca3787b29fe1-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.150528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.150488 5074 generic.go:356] "Generic (PLEG): container finished" podID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerID="3d5d8ff86fc5aae05767dfd4564eb6c6ed29334ee0f85be1d9031636474acf6b" exitCode=0 Oct 31 13:44:17.150725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.150560 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" event={"ID":"fa12a2af-57d5-4270-bd66-2d8d202714cf","Type":"ContainerDied","Data":"3d5d8ff86fc5aae05767dfd4564eb6c6ed29334ee0f85be1d9031636474acf6b"} Oct 31 13:44:17.152220 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.152196 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" Oct 31 13:44:17.152357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.152213 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs" event={"ID":"f25de364-7b78-4e72-b4cb-ca3787b29fe1","Type":"ContainerDied","Data":"e58c6c5db7c3a7599c622d66f475c42a8ca2391d9799d39b920c5d197e198332"} Oct 31 13:44:17.152357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.152264 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e58c6c5db7c3a7599c622d66f475c42a8ca2391d9799d39b920c5d197e198332" Oct 31 13:44:17.258527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.258498 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:44:17.412791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.412747 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-envoy\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.412817 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-kube-api-access-hpxgm\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-kube-api-access-hpxgm\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.412857 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-credential-socket\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.412896 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-token\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-token\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.412926 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-podinfo\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.412987 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa12a2af-57d5-4270-bd66-2d8d202714cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa12a2af-57d5-4270-bd66-2d8d202714cf-istiod-ca-cert\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.413020 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-socket\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.413095 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-data\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.413215 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-credential-socket" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.413758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.413294 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-certs\") pod \"fa12a2af-57d5-4270-bd66-2d8d202714cf\" (UID: \"fa12a2af-57d5-4270-bd66-2d8d202714cf\") " Oct 31 13:44:17.413758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.413549 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa12a2af-57d5-4270-bd66-2d8d202714cf-istiod-ca-cert" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:17.413758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.413745 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-socket" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.413922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.413834 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-data" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.414106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.414086 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-certs" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.414483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.414463 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.414526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.414491 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.414526 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.414507 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa12a2af-57d5-4270-bd66-2d8d202714cf-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.414595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.414522 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.414595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.414536 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.417553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.417522 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-envoy" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.417685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.417595 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-kube-api-access-hpxgm" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "kube-api-access-hpxgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:17.417805 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.417785 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-podinfo" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:17.419111 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.419088 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-token" pod "fa12a2af-57d5-4270-bd66-2d8d202714cf" (UID: "fa12a2af-57d5-4270-bd66-2d8d202714cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:17.421206 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.421185 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v"] Oct 31 13:44:17.430910 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.430888 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x"] Oct 31 13:44:17.515592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.515496 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.515592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.515538 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxgm\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-kube-api-access-hpxgm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.515592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.515557 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.515592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:17.515571 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa12a2af-57d5-4270-bd66-2d8d202714cf-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:18.160586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:18.160551 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" event={"ID":"fa12a2af-57d5-4270-bd66-2d8d202714cf","Type":"ContainerDied","Data":"88773ecb039dd7c456085d79106be87dc5e7869e3abfc3c1a2363b594a5757a0"} Oct 31 13:44:18.160783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:18.160588 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8" Oct 31 13:44:18.160783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:18.160613 5074 scope.go:122] "RemoveContainer" containerID="3d5d8ff86fc5aae05767dfd4564eb6c6ed29334ee0f85be1d9031636474acf6b" Oct 31 13:44:18.195997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:18.195967 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8"] Oct 31 13:44:18.202208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:18.202141 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-569075-openshift-default-6885467b9-zmbt8"] Oct 31 13:44:19.938460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:19.938423 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" path="/var/lib/kubelet/pods/fa12a2af-57d5-4270-bd66-2d8d202714cf/volumes" Oct 31 13:44:20.994954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:20.994927 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:44:21.058135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058057 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-envoy\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.058135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058115 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istiod-ca-cert\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.058446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058172 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-credential-socket\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.058446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058293 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-kube-api-access-hrhls\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-kube-api-access-hrhls\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.058446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058326 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-podinfo\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.058446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058379 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-certs\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.058733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058492 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-data\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.058733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058533 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-socket\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.059483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058871 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-token\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-token\") pod \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\" (UID: \"ec1aec07-a938-44ba-9a2d-e07385a1adf7\") " Oct 31 13:44:21.059483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.058999 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-socket" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.059483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.059043 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-certs" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.059483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.059195 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-data" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.059483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.059426 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-credential-socket" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.059919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.059486 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.059919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.059543 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.059919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.059559 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.059919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.059691 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istiod-ca-cert" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:21.061219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.061187 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-envoy" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.061394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.061370 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-podinfo" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:21.061682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.061658 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-token" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:21.062085 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.062064 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-kube-api-access-hrhls" pod "ec1aec07-a938-44ba-9a2d-e07385a1adf7" (UID: "ec1aec07-a938-44ba-9a2d-e07385a1adf7"). InnerVolumeSpecName "kube-api-access-hrhls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:21.164586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.164299 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.164586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.164346 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.164586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.164364 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1aec07-a938-44ba-9a2d-e07385a1adf7-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.165173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.165057 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrhls\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-kube-api-access-hrhls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.165173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.165129 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.165173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.165149 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1aec07-a938-44ba-9a2d-e07385a1adf7-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.192486 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.192452 5074 generic.go:356] "Generic (PLEG): container finished" podID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerID="1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1" exitCode=0 Oct 31 13:44:21.192705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.192504 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" Oct 31 13:44:21.192705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.192547 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" event={"ID":"ec1aec07-a938-44ba-9a2d-e07385a1adf7","Type":"ContainerDied","Data":"1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1"} Oct 31 13:44:21.192705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.192597 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq" event={"ID":"ec1aec07-a938-44ba-9a2d-e07385a1adf7","Type":"ContainerDied","Data":"86788e1d614a6c8a9a467a7d2fe6a2c062696a1b02ef1f0f70072b921d38d788"} Oct 31 13:44:21.192705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.192643 5074 scope.go:122] "RemoveContainer" containerID="1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1" Oct 31 13:44:21.206947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.206923 5074 scope.go:122] "RemoveContainer" containerID="1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1" Oct 31 13:44:21.207301 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:21.207277 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1\": container with ID starting with 1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1 not found: ID does not exist" containerID="1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1" Oct 31 13:44:21.207391 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.207312 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1"} err="failed to get container status \"1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1\": rpc error: code = NotFound desc = could not find container \"1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1\": container with ID starting with 1c3ca9441b3d9300790b05fda239398aa9efb59edc2c3e21d4d66c75b0bcfad1 not found: ID does not exist" Oct 31 13:44:21.228135 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.228109 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq"] Oct 31 13:44:21.231285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.231261 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5a9e45-openshift-default-7b65f6585c-gdbkq"] Oct 31 13:44:21.939321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:21.939285 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" path="/var/lib/kubelet/pods/ec1aec07-a938-44ba-9a2d-e07385a1adf7/volumes" Oct 31 13:44:24.575894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.575855 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb"] Oct 31 13:44:24.590982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.590946 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4"] Oct 31 13:44:24.966153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.966120 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862"] Oct 31 13:44:24.966939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.966921 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" Oct 31 13:44:24.966996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.966941 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" Oct 31 13:44:24.966996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.966986 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" Oct 31 13:44:24.966996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.966994 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" Oct 31 13:44:24.967105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.967026 5074 cpu_manager.go:414] "containerMap: removing container" podUID="f25de364-7b78-4e72-b4cb-ca3787b29fe1" containerName="revoke-keys" Oct 31 13:44:24.967105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.967034 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25de364-7b78-4e72-b4cb-ca3787b29fe1" containerName="revoke-keys" Oct 31 13:44:24.967170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.967145 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1aec07-a938-44ba-9a2d-e07385a1adf7" containerName="istio-proxy" Oct 31 13:44:24.967170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.967162 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25de364-7b78-4e72-b4cb-ca3787b29fe1" containerName="revoke-keys" Oct 31 13:44:24.967237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.967175 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa12a2af-57d5-4270-bd66-2d8d202714cf" containerName="istio-proxy" Oct 31 13:44:24.971091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.971072 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:24.980059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.980036 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:44:24.980399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.980378 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:44:24.986385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.986360 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:44:24.986491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.986415 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:44:24.990092 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:24.990062 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862"] Oct 31 13:44:25.018753 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.018711 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.019022 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.018992 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljt4\" (UniqueName: \"kubernetes.io/projected/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-kube-api-access-sljt4\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.019107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.019072 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.120490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.120457 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.120708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.120574 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljt4\" (UniqueName: \"kubernetes.io/projected/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-kube-api-access-sljt4\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.120708 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.120613 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.121358 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.121332 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.123177 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.123154 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.145669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.145611 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljt4\" (UniqueName: \"kubernetes.io/projected/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-kube-api-access-sljt4\") pod \"maas-api-revoke-sub-50523673-dz862\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.286621 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.286541 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:25.593409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:25.593372 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862"] Oct 31 13:44:25.595033 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:44:25.595002 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77aff0fd_d5b7_4943_bcac_2a59bd0a66fc.slice/crio-066452ac466b185bb0fc9b2b87f993bbbb12c026bbd065ef3e717f76e3e1c27a WatchSource:0}: Error finding container 066452ac466b185bb0fc9b2b87f993bbbb12c026bbd065ef3e717f76e3e1c27a: Status 404 returned error can't find the container with id 066452ac466b185bb0fc9b2b87f993bbbb12c026bbd065ef3e717f76e3e1c27a Oct 31 13:44:25.985921 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:25.985882 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:44:26.243221 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:26.243180 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" event={"ID":"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc","Type":"ContainerStarted","Data":"9f9dea499bff455cec53ffaee4ad5e1728bf7f528b77921af3d622caebb2ad05"} Oct 31 13:44:26.243425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:26.243240 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" event={"ID":"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc","Type":"ContainerStarted","Data":"066452ac466b185bb0fc9b2b87f993bbbb12c026bbd065ef3e717f76e3e1c27a"} Oct 31 13:44:26.285905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:26.285852 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" podStartSLOduration=2.285836855 podStartE2EDuration="2.285836855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:26.284841024 +0000 UTC m=+4360.462559025" watchObservedRunningTime="2026-10-31 13:44:26.285836855 +0000 UTC m=+4360.463554833" Oct 31 13:44:27.252005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:27.251970 5074 generic.go:356] "Generic (PLEG): container finished" podID="77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" containerID="9f9dea499bff455cec53ffaee4ad5e1728bf7f528b77921af3d622caebb2ad05" exitCode=0 Oct 31 13:44:27.252465 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:27.252056 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" event={"ID":"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc","Type":"ContainerDied","Data":"9f9dea499bff455cec53ffaee4ad5e1728bf7f528b77921af3d622caebb2ad05"} Oct 31 13:44:28.413299 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.413272 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:28.579146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.579048 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-openshift-ca\") pod \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " Oct 31 13:44:28.579146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.579144 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-kube-api-access-sljt4\" (UniqueName: \"kubernetes.io/projected/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-kube-api-access-sljt4\") pod \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " Oct 31 13:44:28.579402 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.579289 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-xks-ca\") pod \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\" (UID: \"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc\") " Oct 31 13:44:28.579480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.579455 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-openshift-ca" pod "77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" (UID: "77aff0fd-d5b7-4943-bcac-2a59bd0a66fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:28.579782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.579757 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.581395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.581362 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-kube-api-access-sljt4" pod "77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" (UID: "77aff0fd-d5b7-4943-bcac-2a59bd0a66fc"). InnerVolumeSpecName "kube-api-access-sljt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:28.581491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.581402 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-xks-ca" pod "77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" (UID: "77aff0fd-d5b7-4943-bcac-2a59bd0a66fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:28.681782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.681739 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:28.681782 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:28.681783 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljt4\" (UniqueName: \"kubernetes.io/projected/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc-kube-api-access-sljt4\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:29.270937 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:29.270906 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" Oct 31 13:44:29.271667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:29.270952 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862" event={"ID":"77aff0fd-d5b7-4943-bcac-2a59bd0a66fc","Type":"ContainerDied","Data":"066452ac466b185bb0fc9b2b87f993bbbb12c026bbd065ef3e717f76e3e1c27a"} Oct 31 13:44:29.271768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:29.271705 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066452ac466b185bb0fc9b2b87f993bbbb12c026bbd065ef3e717f76e3e1c27a" Oct 31 13:44:29.576648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:29.576538 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" podUID="96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" containerName="payload-processing" containerID="cri-o://459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d" gracePeriod=25 Oct 31 13:44:29.592191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:29.592137 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" podUID="d42989ca-a6d6-4900-94ee-269a7c92ed94" containerName="payload-pre-processing" containerID="cri-o://0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d" gracePeriod=25 Oct 31 13:44:29.890768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:29.890736 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 13:44:29.906573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:29.906548 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 13:44:30.008574 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.008537 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-plugins-config-volume\") pod \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " Oct 31 13:44:30.008779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.008683 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42989ca-a6d6-4900-94ee-269a7c92ed94-kube-api-access-9jvb8\" (UniqueName: \"kubernetes.io/projected/d42989ca-a6d6-4900-94ee-269a7c92ed94-kube-api-access-9jvb8\") pod \"d42989ca-a6d6-4900-94ee-269a7c92ed94\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " Oct 31 13:44:30.008779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.008710 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-kube-api-access-ccc2m\" (UniqueName: \"kubernetes.io/projected/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-kube-api-access-ccc2m\") pod \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\" (UID: \"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7\") " Oct 31 13:44:30.008927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.008873 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42989ca-a6d6-4900-94ee-269a7c92ed94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42989ca-a6d6-4900-94ee-269a7c92ed94-plugins-config-volume\") pod \"d42989ca-a6d6-4900-94ee-269a7c92ed94\" (UID: \"d42989ca-a6d6-4900-94ee-269a7c92ed94\") " Oct 31 13:44:30.009079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.009046 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-plugins-config-volume" pod "96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" (UID: "96ad5b2c-ee99-4d37-8ff0-4025ee6426b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:30.009301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.009270 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42989ca-a6d6-4900-94ee-269a7c92ed94-plugins-config-volume" pod "d42989ca-a6d6-4900-94ee-269a7c92ed94" (UID: "d42989ca-a6d6-4900-94ee-269a7c92ed94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:30.009467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.009444 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d42989ca-a6d6-4900-94ee-269a7c92ed94-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:30.009530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.009474 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:30.011120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.011095 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-kube-api-access-ccc2m" pod "96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" (UID: "96ad5b2c-ee99-4d37-8ff0-4025ee6426b7"). InnerVolumeSpecName "kube-api-access-ccc2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:30.011303 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.011286 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42989ca-a6d6-4900-94ee-269a7c92ed94-kube-api-access-9jvb8" pod "d42989ca-a6d6-4900-94ee-269a7c92ed94" (UID: "d42989ca-a6d6-4900-94ee-269a7c92ed94"). InnerVolumeSpecName "kube-api-access-9jvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:30.111081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.110979 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvb8\" (UniqueName: \"kubernetes.io/projected/d42989ca-a6d6-4900-94ee-269a7c92ed94-kube-api-access-9jvb8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:30.111081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.111011 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccc2m\" (UniqueName: \"kubernetes.io/projected/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7-kube-api-access-ccc2m\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:30.282171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.282132 5074 generic.go:356] "Generic (PLEG): container finished" podID="d42989ca-a6d6-4900-94ee-269a7c92ed94" containerID="0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d" exitCode=0 Oct 31 13:44:30.282171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.282180 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" Oct 31 13:44:30.282446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.282238 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" event={"ID":"d42989ca-a6d6-4900-94ee-269a7c92ed94","Type":"ContainerDied","Data":"0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d"} Oct 31 13:44:30.282446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.282311 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4" event={"ID":"d42989ca-a6d6-4900-94ee-269a7c92ed94","Type":"ContainerDied","Data":"ff2cbe8a1fb618a9261da19ee283e868266742451afdda08f491ff2b43542048"} Oct 31 13:44:30.282446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.282342 5074 scope.go:122] "RemoveContainer" containerID="0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d" Oct 31 13:44:30.283934 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.283911 5074 generic.go:356] "Generic (PLEG): container finished" podID="96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" containerID="459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d" exitCode=0 Oct 31 13:44:30.284675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.284616 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" Oct 31 13:44:30.286327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.284104 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" event={"ID":"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7","Type":"ContainerDied","Data":"459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d"} Oct 31 13:44:30.286327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.286039 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb" event={"ID":"96ad5b2c-ee99-4d37-8ff0-4025ee6426b7","Type":"ContainerDied","Data":"eb073e40d8dc5c9fbfb35288842bf25bf6499a0bdf84972a15391b3dd750e720"} Oct 31 13:44:30.314249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.314149 5074 scope.go:122] "RemoveContainer" containerID="0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d" Oct 31 13:44:30.316991 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:30.316960 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d\": container with ID starting with 0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d not found: ID does not exist" containerID="0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d" Oct 31 13:44:30.317152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.316999 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d"} err="failed to get container status \"0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d\": rpc error: code = NotFound desc = could not find container \"0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d\": container with ID starting with 0db78e5f084514e29643d2b4d827f49661d8dfadad8035c1591175a85e7b962d not found: ID does not exist" Oct 31 13:44:30.317152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.317021 5074 scope.go:122] "RemoveContainer" containerID="459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d" Oct 31 13:44:30.325043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.325018 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4"] Oct 31 13:44:30.328405 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.328382 5074 scope.go:122] "RemoveContainer" containerID="459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d" Oct 31 13:44:30.328741 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:30.328717 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d\": container with ID starting with 459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d not found: ID does not exist" containerID="459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d" Oct 31 13:44:30.328830 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.328747 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d"} err="failed to get container status \"459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d\": rpc error: code = NotFound desc = could not find container \"459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d\": container with ID starting with 459a228fb0bf6e83acef7fe4a19799299987a8682f413b2cc39b5bc8f5fce64d not found: ID does not exist" Oct 31 13:44:30.330792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.330773 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3dd1e5-6bdd8865d7-9l9s4"] Oct 31 13:44:30.343773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.343749 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb"] Oct 31 13:44:30.347505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:30.347481 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3dd1e5-7b97f7fc6c-97rwb"] Oct 31 13:44:31.938163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:31.938119 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" path="/var/lib/kubelet/pods/96ad5b2c-ee99-4d37-8ff0-4025ee6426b7/volumes" Oct 31 13:44:31.938725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:31.938700 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42989ca-a6d6-4900-94ee-269a7c92ed94" path="/var/lib/kubelet/pods/d42989ca-a6d6-4900-94ee-269a7c92ed94/volumes" Oct 31 13:44:32.442860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.442813 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" containerID="cri-o://e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72" gracePeriod=45 Oct 31 13:44:32.451356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.451318 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" containerID="cri-o://3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a" gracePeriod=45 Oct 31 13:44:32.484242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.483134 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:44:32.751759 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.751733 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 13:44:32.755047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.755027 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 13:44:32.945915 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.945879 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tls-certs\" (UniqueName: \"kubernetes.io/secret/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tls-certs\") pod \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.945935 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c900e1-3941-45ce-b10f-d892634d7596-kube-api-access-s6qkf\" (UniqueName: \"kubernetes.io/projected/19c900e1-3941-45ce-b10f-d892634d7596-kube-api-access-s6qkf\") pod \"19c900e1-3941-45ce-b10f-d892634d7596\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946001 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tmp-dir\") pod \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946040 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-model-cache\") pod \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946092 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-home\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-home\") pod \"19c900e1-3941-45ce-b10f-d892634d7596\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946150 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c900e1-3941-45ce-b10f-d892634d7596-tls-certs\" (UniqueName: \"kubernetes.io/secret/19c900e1-3941-45ce-b10f-d892634d7596-tls-certs\") pod \"19c900e1-3941-45ce-b10f-d892634d7596\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946199 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-model-cache\") pod \"19c900e1-3941-45ce-b10f-d892634d7596\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946249 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-home\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-home\") pod \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946276 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-dshm\") pod \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " Oct 31 13:44:32.946403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946324 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-tmp-dir\") pod \"19c900e1-3941-45ce-b10f-d892634d7596\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " Oct 31 13:44:32.946949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946410 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-dshm\") pod \"19c900e1-3941-45ce-b10f-d892634d7596\" (UID: \"19c900e1-3941-45ce-b10f-d892634d7596\") " Oct 31 13:44:32.946949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946481 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-model-cache" pod "85a5c9b7-eada-4d26-8e3a-e07bb9215c53" (UID: "85a5c9b7-eada-4d26-8e3a-e07bb9215c53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.946949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946486 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-kube-api-access-7d89d\" (UniqueName: \"kubernetes.io/projected/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-kube-api-access-7d89d\") pod \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\" (UID: \"85a5c9b7-eada-4d26-8e3a-e07bb9215c53\") " Oct 31 13:44:32.946949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946582 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-model-cache" pod "19c900e1-3941-45ce-b10f-d892634d7596" (UID: "19c900e1-3941-45ce-b10f-d892634d7596"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.946949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946605 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-home" pod "85a5c9b7-eada-4d26-8e3a-e07bb9215c53" (UID: "85a5c9b7-eada-4d26-8e3a-e07bb9215c53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.946949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946729 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-home" pod "19c900e1-3941-45ce-b10f-d892634d7596" (UID: "19c900e1-3941-45ce-b10f-d892634d7596"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.946949 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946749 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-tmp-dir" pod "19c900e1-3941-45ce-b10f-d892634d7596" (UID: "19c900e1-3941-45ce-b10f-d892634d7596"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.947351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.946425 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tmp-dir" pod "85a5c9b7-eada-4d26-8e3a-e07bb9215c53" (UID: "85a5c9b7-eada-4d26-8e3a-e07bb9215c53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.947351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.947336 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.947473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.947358 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.947473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.947374 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.947473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.947391 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.947473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.947405 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.947473 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.947419 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:32.948691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.948667 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tls-certs" pod "85a5c9b7-eada-4d26-8e3a-e07bb9215c53" (UID: "85a5c9b7-eada-4d26-8e3a-e07bb9215c53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:32.948840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.948809 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c900e1-3941-45ce-b10f-d892634d7596-kube-api-access-s6qkf" pod "19c900e1-3941-45ce-b10f-d892634d7596" (UID: "19c900e1-3941-45ce-b10f-d892634d7596"). InnerVolumeSpecName "kube-api-access-s6qkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:32.949082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.949037 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c900e1-3941-45ce-b10f-d892634d7596-tls-certs" pod "19c900e1-3941-45ce-b10f-d892634d7596" (UID: "19c900e1-3941-45ce-b10f-d892634d7596"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:32.949281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.949229 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-kube-api-access-7d89d" pod "85a5c9b7-eada-4d26-8e3a-e07bb9215c53" (UID: "85a5c9b7-eada-4d26-8e3a-e07bb9215c53"). InnerVolumeSpecName "kube-api-access-7d89d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:32.949722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.949696 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-dshm" pod "19c900e1-3941-45ce-b10f-d892634d7596" (UID: "19c900e1-3941-45ce-b10f-d892634d7596"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:32.949722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:32.949702 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-dshm" pod "85a5c9b7-eada-4d26-8e3a-e07bb9215c53" (UID: "85a5c9b7-eada-4d26-8e3a-e07bb9215c53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:33.049356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.049251 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c900e1-3941-45ce-b10f-d892634d7596-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.049356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.049299 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.049356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.049313 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c900e1-3941-45ce-b10f-d892634d7596-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.049356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.049330 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d89d\" (UniqueName: \"kubernetes.io/projected/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-kube-api-access-7d89d\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.049356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.049344 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a5c9b7-eada-4d26-8e3a-e07bb9215c53-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.049356 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.049358 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6qkf\" (UniqueName: \"kubernetes.io/projected/19c900e1-3941-45ce-b10f-d892634d7596-kube-api-access-s6qkf\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:33.318573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.318197 5074 generic.go:356] "Generic (PLEG): container finished" podID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerID="3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a" exitCode=0 Oct 31 13:44:33.318573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.318267 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" Oct 31 13:44:33.318573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.318318 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" event={"ID":"85a5c9b7-eada-4d26-8e3a-e07bb9215c53","Type":"ContainerDied","Data":"3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a"} Oct 31 13:44:33.318573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.318358 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x" event={"ID":"85a5c9b7-eada-4d26-8e3a-e07bb9215c53","Type":"ContainerDied","Data":"e8e490a34534e36f3e2f11b36d936ede11af20c77de5e96e0aac0672f479085a"} Oct 31 13:44:33.318573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.318380 5074 scope.go:122] "RemoveContainer" containerID="3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a" Oct 31 13:44:33.326609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.326427 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" Oct 31 13:44:33.328553 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.326461 5074 generic.go:356] "Generic (PLEG): container finished" podID="19c900e1-3941-45ce-b10f-d892634d7596" containerID="e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72" exitCode=0 Oct 31 13:44:33.328955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.326485 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" event={"ID":"19c900e1-3941-45ce-b10f-d892634d7596","Type":"ContainerDied","Data":"e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72"} Oct 31 13:44:33.329057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.328978 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v" event={"ID":"19c900e1-3941-45ce-b10f-d892634d7596","Type":"ContainerDied","Data":"5a14b8f43dd771030a2dff9c4cadbb42734a57de67e4c4b2f475ef0adbd98351"} Oct 31 13:44:33.355011 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.354977 5074 scope.go:122] "RemoveContainer" containerID="3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a" Oct 31 13:44:33.355623 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:33.355585 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a\": container with ID starting with 3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a not found: ID does not exist" containerID="3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a" Oct 31 13:44:33.356087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.356016 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a"} err="failed to get container status \"3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a\": rpc error: code = NotFound desc = could not find container \"3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a\": container with ID starting with 3daa323f361ffd43b3c9ccb3120972dc7a6629d3fbb3b156f71381a27769ec3a not found: ID does not exist" Oct 31 13:44:33.356143 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.356093 5074 scope.go:122] "RemoveContainer" containerID="e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72" Oct 31 13:44:33.366690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.366672 5074 scope.go:122] "RemoveContainer" containerID="e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72" Oct 31 13:44:33.367121 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:33.367096 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72\": container with ID starting with e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72 not found: ID does not exist" containerID="e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72" Oct 31 13:44:33.367236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.367217 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72"} err="failed to get container status \"e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72\": rpc error: code = NotFound desc = could not find container \"e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72\": container with ID starting with e453064d82292a4c117136658e8f4115f1bf8a9cbe36b89c28aac815265a3b72 not found: ID does not exist" Oct 31 13:44:33.413646 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.413596 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x"] Oct 31 13:44:33.416674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.416610 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/simulator-w2-3dd1e5-kserve-6569f94675-2884x"] Oct 31 13:44:33.442195 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.442163 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v"] Oct 31 13:44:33.448469 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.448440 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3dd1e5/premium-w2-3dd1e5-kserve-7c454c44fd-ptl8v"] Oct 31 13:44:33.941252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.941222 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c900e1-3941-45ce-b10f-d892634d7596" path="/var/lib/kubelet/pods/19c900e1-3941-45ce-b10f-d892634d7596/volumes" Oct 31 13:44:33.941734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:33.941719 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" path="/var/lib/kubelet/pods/85a5c9b7-eada-4d26-8e3a-e07bb9215c53/volumes" Oct 31 13:44:36.013847 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:36.013803 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f"] Oct 31 13:44:36.014273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:36.014081 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" containerID="cri-o://e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d" gracePeriod=30 Oct 31 13:44:41.303611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.303543 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:44:41.392302 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.392261 5074 generic.go:356] "Generic (PLEG): container finished" podID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerID="e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d" exitCode=0 Oct 31 13:44:41.392488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.392321 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" Oct 31 13:44:41.392488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.392360 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" event={"ID":"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd","Type":"ContainerDied","Data":"e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d"} Oct 31 13:44:41.392488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.392439 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" event={"ID":"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd","Type":"ContainerDied","Data":"7442a1387646a6c31b7e0a773c9824b6fa846bb25487cdfccc0081c7e424d04e"} Oct 31 13:44:41.392488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.392474 5074 scope.go:122] "RemoveContainer" containerID="e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d" Oct 31 13:44:41.403668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.403650 5074 scope.go:122] "RemoveContainer" containerID="e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d" Oct 31 13:44:41.403980 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:41.403940 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d\": container with ID starting with e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d not found: ID does not exist" containerID="e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d" Oct 31 13:44:41.404105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.403981 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d"} err="failed to get container status \"e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d\": rpc error: code = NotFound desc = could not find container \"e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d\": container with ID starting with e519cf6e3103e0b0fedc306231ecd3f091a3be585eb13de08364cd261bf74d8d not found: ID does not exist" Oct 31 13:44:41.405484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405461 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istiod-ca-cert\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.405570 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405506 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-podinfo\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.405627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405587 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-token\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-token\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.405786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405695 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-socket\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.405786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405729 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-envoy\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.405909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405792 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-kube-api-access-8smkq\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-kube-api-access-8smkq\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.405909 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405896 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-credential-socket\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.406038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405905 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istiod-ca-cert" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:41.406038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405935 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-certs\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.406038 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.405977 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-data\") pod \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\" (UID: \"a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd\") " Oct 31 13:44:41.406219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.406178 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-socket" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:41.406409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.406385 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-data" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:41.406508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.406462 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.406508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.406481 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.406720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.406699 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-credential-socket" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:41.406922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.406871 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-certs" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:41.408863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.408793 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-podinfo" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:41.408863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.408826 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-envoy" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:41.409021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.408999 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-kube-api-access-8smkq" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "kube-api-access-8smkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:41.410013 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.409985 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-token" pod "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" (UID: "a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:41.508310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.508272 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.508310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.508311 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smkq\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-kube-api-access-8smkq\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.508603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.508325 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.508603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.508338 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.508603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.508351 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.508603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.508362 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.508603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.508374 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:41.721312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.721277 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f"] Oct 31 13:44:41.724390 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.724358 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f"] Oct 31 13:44:41.938277 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:41.938231 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" path="/var/lib/kubelet/pods/a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd/volumes" Oct 31 13:44:42.221740 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:42.221694 5074 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:44:42.221946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:42.221775 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-4574ec-openshift-default-5b57bb4769-klr5f" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:44:42.236260 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:42.236221 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr"] Oct 31 13:44:46.723489 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.723447 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff"] Oct 31 13:44:46.724127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724105 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724127 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724146 5074 cpu_manager.go:414] "containerMap: removing container" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724154 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724168 5074 cpu_manager.go:414] "containerMap: removing container" podUID="96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" containerName="payload-processing" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724176 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" containerName="payload-processing" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724216 5074 cpu_manager.go:414] "containerMap: removing container" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724227 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724245 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d42989ca-a6d6-4900-94ee-269a7c92ed94" containerName="payload-pre-processing" Oct 31 13:44:46.724240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724254 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42989ca-a6d6-4900-94ee-269a7c92ed94" containerName="payload-pre-processing" Oct 31 13:44:46.724788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724282 5074 cpu_manager.go:414] "containerMap: removing container" podUID="77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" containerName="revoke-keys" Oct 31 13:44:46.724788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724290 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" containerName="revoke-keys" Oct 31 13:44:46.724788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724475 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e86b90-dc70-4ef8-9ff5-fcfd8c9970bd" containerName="istio-proxy" Oct 31 13:44:46.724788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724490 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c900e1-3941-45ce-b10f-d892634d7596" containerName="main" Oct 31 13:44:46.724980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724804 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ad5b2c-ee99-4d37-8ff0-4025ee6426b7" containerName="payload-processing" Oct 31 13:44:46.724980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724821 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a5c9b7-eada-4d26-8e3a-e07bb9215c53" containerName="main" Oct 31 13:44:46.724980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724834 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42989ca-a6d6-4900-94ee-269a7c92ed94" containerName="payload-pre-processing" Oct 31 13:44:46.724980 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.724847 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" containerName="revoke-keys" Oct 31 13:44:46.730193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.730164 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff"] Oct 31 13:44:46.730334 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.730262 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.744652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.736900 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:44:46.744652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.738184 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:44:46.744652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.738684 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:44:46.744652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.738859 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:44:46.878085 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.878052 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.878269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.878104 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.878269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.878200 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rm2\" (UniqueName: \"kubernetes.io/projected/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-kube-api-access-d4rm2\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.979898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.979818 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.979898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.979887 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.980171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.979946 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rm2\" (UniqueName: \"kubernetes.io/projected/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-kube-api-access-d4rm2\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.980722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.980692 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.982552 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.982517 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:46.993550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:46.993519 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rm2\" (UniqueName: \"kubernetes.io/projected/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-kube-api-access-d4rm2\") pod \"maas-api-revoke-sub-49a5aebf-bjzff\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:47.051494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:47.051461 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:47.237495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:47.237387 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" podUID="d9ab647a-b861-4530-8d48-f9a05832887b" containerName="payload-processing" containerID="cri-o://60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a" gracePeriod=25 Oct 31 13:44:47.348724 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:47.348695 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff"] Oct 31 13:44:47.349681 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:44:47.349649 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5ec8be_e55f_4517_a5d1_03c283bc2c69.slice/crio-9af6cfb50ded93265fd7f3f97eed0048896000e50b87a44d695f7b65fe0949d5 WatchSource:0}: Error finding container 9af6cfb50ded93265fd7f3f97eed0048896000e50b87a44d695f7b65fe0949d5: Status 404 returned error can't find the container with id 9af6cfb50ded93265fd7f3f97eed0048896000e50b87a44d695f7b65fe0949d5 Oct 31 13:44:47.439939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:47.439907 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" event={"ID":"ac5ec8be-e55f-4517-a5d1-03c283bc2c69","Type":"ContainerStarted","Data":"9af6cfb50ded93265fd7f3f97eed0048896000e50b87a44d695f7b65fe0949d5"} Oct 31 13:44:48.448384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:48.448352 5074 generic.go:356] "Generic (PLEG): container finished" podID="ac5ec8be-e55f-4517-a5d1-03c283bc2c69" containerID="82d304c4323067b910dcbcd5a7c637b1ed33230ef209599634a760e239b4b0f2" exitCode=0 Oct 31 13:44:48.448798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:48.448399 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" event={"ID":"ac5ec8be-e55f-4517-a5d1-03c283bc2c69","Type":"ContainerDied","Data":"82d304c4323067b910dcbcd5a7c637b1ed33230ef209599634a760e239b4b0f2"} Oct 31 13:44:49.641327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.641298 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:49.744573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.740586 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-openshift-ca\") pod \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " Oct 31 13:44:49.744573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.740832 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-xks-ca\") pod \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " Oct 31 13:44:49.744573 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.740878 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-kube-api-access-d4rm2\" (UniqueName: \"kubernetes.io/projected/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-kube-api-access-d4rm2\") pod \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\" (UID: \"ac5ec8be-e55f-4517-a5d1-03c283bc2c69\") " Oct 31 13:44:49.752825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.752784 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-kube-api-access-d4rm2" pod "ac5ec8be-e55f-4517-a5d1-03c283bc2c69" (UID: "ac5ec8be-e55f-4517-a5d1-03c283bc2c69"). InnerVolumeSpecName "kube-api-access-d4rm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:49.756073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.756018 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-openshift-ca" pod "ac5ec8be-e55f-4517-a5d1-03c283bc2c69" (UID: "ac5ec8be-e55f-4517-a5d1-03c283bc2c69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:49.784917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.784850 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-xks-ca" pod "ac5ec8be-e55f-4517-a5d1-03c283bc2c69" (UID: "ac5ec8be-e55f-4517-a5d1-03c283bc2c69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:49.842544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.842506 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:49.842544 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.842544 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rm2\" (UniqueName: \"kubernetes.io/projected/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-kube-api-access-d4rm2\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:49.842776 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:49.842560 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5ec8be-e55f-4517-a5d1-03c283bc2c69-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:50.002817 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:50.002750 5074 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5ec8be_e55f_4517_a5d1_03c283bc2c69.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:44:50.467609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:50.467572 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" event={"ID":"ac5ec8be-e55f-4517-a5d1-03c283bc2c69","Type":"ContainerDied","Data":"9af6cfb50ded93265fd7f3f97eed0048896000e50b87a44d695f7b65fe0949d5"} Oct 31 13:44:50.467609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:50.467603 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff" Oct 31 13:44:50.467838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:50.467624 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af6cfb50ded93265fd7f3f97eed0048896000e50b87a44d695f7b65fe0949d5" Oct 31 13:44:52.225294 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:52.225256 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" podUID="d9ab647a-b861-4530-8d48-f9a05832887b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:44:53.453200 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:53.453163 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr"] Oct 31 13:44:53.453834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:53.453430 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" containerID="cri-o://4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186" gracePeriod=30 Oct 31 13:44:54.368208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.368170 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh"] Oct 31 13:44:54.368710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.368686 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ac5ec8be-e55f-4517-a5d1-03c283bc2c69" containerName="revoke-keys" Oct 31 13:44:54.368710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.368703 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5ec8be-e55f-4517-a5d1-03c283bc2c69" containerName="revoke-keys" Oct 31 13:44:54.368882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.368812 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5ec8be-e55f-4517-a5d1-03c283bc2c69" containerName="revoke-keys" Oct 31 13:44:54.372653 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.372622 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.385002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.383684 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:44:54.385002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.383817 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:44:54.387043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.387017 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh"] Oct 31 13:44:54.390269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.390243 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:44:54.390538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.390520 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:44:54.501611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.501562 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.502150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.501748 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hc5\" (UniqueName: \"kubernetes.io/projected/b7c98812-e834-4f7e-91f4-41a22574c8e4-kube-api-access-26hc5\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.502150 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.501866 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.602890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.602850 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.603095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.602981 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hc5\" (UniqueName: \"kubernetes.io/projected/b7c98812-e834-4f7e-91f4-41a22574c8e4-kube-api-access-26hc5\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.603095 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.603071 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.603627 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.603595 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.605668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.605605 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.617589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.617567 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hc5\" (UniqueName: \"kubernetes.io/projected/b7c98812-e834-4f7e-91f4-41a22574c8e4-kube-api-access-26hc5\") pod \"maas-api-revoke-sub-826c638f-cfrdh\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:54.700054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.699955 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:55.000019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:54.999947 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh"] Oct 31 13:44:55.232584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:55.231558 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7"] Oct 31 13:44:55.232584 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:55.231821 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" containerID="cri-o://e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7" gracePeriod=30 Oct 31 13:44:55.508088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:55.507975 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" event={"ID":"b7c98812-e834-4f7e-91f4-41a22574c8e4","Type":"ContainerStarted","Data":"443b8e71a585df6b4d9fd6d03dc18184829650b6e2d92ece8248530c97813134"} Oct 31 13:44:55.508088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:55.508044 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" event={"ID":"b7c98812-e834-4f7e-91f4-41a22574c8e4","Type":"ContainerStarted","Data":"b14a6caa366ab63185e274576dc64f1e18c5e4244833712ebb4cfc2c2cf8ce55"} Oct 31 13:44:55.538483 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:55.538380 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" podStartSLOduration=1.538366616 podStartE2EDuration="1.538366616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:55.537253588 +0000 UTC m=+4389.714971569" watchObservedRunningTime="2026-10-31 13:44:55.538366616 +0000 UTC m=+4389.716084602" Oct 31 13:44:55.986401 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:55.986367 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:44:56.517742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:56.517704 5074 generic.go:356] "Generic (PLEG): container finished" podID="b7c98812-e834-4f7e-91f4-41a22574c8e4" containerID="443b8e71a585df6b4d9fd6d03dc18184829650b6e2d92ece8248530c97813134" exitCode=0 Oct 31 13:44:56.518198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:56.517828 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" event={"ID":"b7c98812-e834-4f7e-91f4-41a22574c8e4","Type":"ContainerDied","Data":"443b8e71a585df6b4d9fd6d03dc18184829650b6e2d92ece8248530c97813134"} Oct 31 13:44:56.537005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:56.536979 5074 scope.go:122] "RemoveContainer" containerID="493cd6121b8f8bf9380ffe15576c3eb7944986dc44768dabb07aa1704f543371" Oct 31 13:44:56.685341 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:56.685308 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:44:57.048296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.048262 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59"] Oct 31 13:44:57.053862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.053842 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.089445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.089414 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59"] Oct 31 13:44:57.141454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.141417 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.141665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.141469 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.141665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.141613 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9h2s\" (UniqueName: \"kubernetes.io/projected/7842b9dc-893b-4c00-90b2-8283e5b763b3-kube-api-access-n9h2s\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.242700 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.242660 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.242897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.242758 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9h2s\" (UniqueName: \"kubernetes.io/projected/7842b9dc-893b-4c00-90b2-8283e5b763b3-kube-api-access-n9h2s\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.242940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.242909 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.243450 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.243418 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.245241 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.245212 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.258490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.258469 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9h2s\" (UniqueName: \"kubernetes.io/projected/7842b9dc-893b-4c00-90b2-8283e5b763b3-kube-api-access-n9h2s\") pod \"maas-api-revoke-sub-e919951f-djv59\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.368000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.367922 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:44:57.654773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.654743 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59"] Oct 31 13:44:57.674271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.674242 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:57.748414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.748375 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-xks-ca\") pod \"b7c98812-e834-4f7e-91f4-41a22574c8e4\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " Oct 31 13:44:57.748622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.748423 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-openshift-ca\") pod \"b7c98812-e834-4f7e-91f4-41a22574c8e4\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " Oct 31 13:44:57.748622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.748467 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c98812-e834-4f7e-91f4-41a22574c8e4-kube-api-access-26hc5\" (UniqueName: \"kubernetes.io/projected/b7c98812-e834-4f7e-91f4-41a22574c8e4-kube-api-access-26hc5\") pod \"b7c98812-e834-4f7e-91f4-41a22574c8e4\" (UID: \"b7c98812-e834-4f7e-91f4-41a22574c8e4\") " Oct 31 13:44:57.748953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.748920 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-openshift-ca" pod "b7c98812-e834-4f7e-91f4-41a22574c8e4" (UID: "b7c98812-e834-4f7e-91f4-41a22574c8e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:57.750488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.750462 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-xks-ca" pod "b7c98812-e834-4f7e-91f4-41a22574c8e4" (UID: "b7c98812-e834-4f7e-91f4-41a22574c8e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:57.751178 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.751148 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c98812-e834-4f7e-91f4-41a22574c8e4-kube-api-access-26hc5" pod "b7c98812-e834-4f7e-91f4-41a22574c8e4" (UID: "b7c98812-e834-4f7e-91f4-41a22574c8e4"). InnerVolumeSpecName "kube-api-access-26hc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:57.850357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.850327 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:57.850357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.850355 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c98812-e834-4f7e-91f4-41a22574c8e4-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:57.850541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:57.850366 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hc5\" (UniqueName: \"kubernetes.io/projected/b7c98812-e834-4f7e-91f4-41a22574c8e4-kube-api-access-26hc5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.532545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.532453 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" event={"ID":"7842b9dc-893b-4c00-90b2-8283e5b763b3","Type":"ContainerStarted","Data":"49b86c887b8074a7e83dd62a5efc6cd9a1207ade87cb11c640fa095e16679079"} Oct 31 13:44:58.532545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.532522 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" event={"ID":"7842b9dc-893b-4c00-90b2-8283e5b763b3","Type":"ContainerStarted","Data":"d8f65ca387a8ea799de62d35e027d981503d0cc4f7f45774698d1a71a4656bd7"} Oct 31 13:44:58.533959 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.533934 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" Oct 31 13:44:58.534067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.533979 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh" event={"ID":"b7c98812-e834-4f7e-91f4-41a22574c8e4","Type":"ContainerDied","Data":"b14a6caa366ab63185e274576dc64f1e18c5e4244833712ebb4cfc2c2cf8ce55"} Oct 31 13:44:58.534067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.534023 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14a6caa366ab63185e274576dc64f1e18c5e4244833712ebb4cfc2c2cf8ce55" Oct 31 13:44:58.563797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.563744 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" podStartSLOduration=2.563724654 podStartE2EDuration="2.563724654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:58.560627949 +0000 UTC m=+4392.738345933" watchObservedRunningTime="2026-10-31 13:44:58.563724654 +0000 UTC m=+4392.741442639" Oct 31 13:44:58.796392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.796304 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:44:58.867816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.867776 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-kube-api-access-v86w7\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-kube-api-access-v86w7\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.868527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.868446 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-data\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.868696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.868548 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istiod-ca-cert\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.869073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869012 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-data" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:58.869073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869047 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-podinfo\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.869270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869125 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-envoy\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.871860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869276 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-socket\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.871860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869324 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-certs\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.871860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869384 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-token\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.871860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869430 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-credential-socket\") pod \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\" (UID: \"8ff51ea1-b218-4ca6-ac95-d81bdabe324f\") " Oct 31 13:44:58.871860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869980 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.872252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.869017 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istiod-ca-cert" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:58.872252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.870268 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-credential-socket" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:58.872252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.870450 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-socket" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:58.872252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.870622 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-certs" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:58.872252 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.872204 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-envoy" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:58.873990 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.873943 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-podinfo" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:58.875884 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.875159 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-kube-api-access-v86w7" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "kube-api-access-v86w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:58.876267 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.876232 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-token" pod "8ff51ea1-b218-4ca6-ac95-d81bdabe324f" (UID: "8ff51ea1-b218-4ca6-ac95-d81bdabe324f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:58.921961 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.921923 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862"] Oct 31 13:44:58.942579 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.942545 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg"] Oct 31 13:44:58.951771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.951743 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv"] Oct 31 13:44:58.966123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.966082 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc"] Oct 31 13:44:58.971791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971760 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.971791 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971793 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.971996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971808 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.971996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971821 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.971996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971835 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86w7\" (UniqueName: \"kubernetes.io/projected/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-kube-api-access-v86w7\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.971996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971850 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.971996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971863 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:58.971996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:58.971877 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff51ea1-b218-4ca6-ac95-d81bdabe324f-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.542184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.542091 5074 generic.go:356] "Generic (PLEG): container finished" podID="7842b9dc-893b-4c00-90b2-8283e5b763b3" containerID="49b86c887b8074a7e83dd62a5efc6cd9a1207ade87cb11c640fa095e16679079" exitCode=0 Oct 31 13:44:59.542184 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.542170 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" event={"ID":"7842b9dc-893b-4c00-90b2-8283e5b763b3","Type":"ContainerDied","Data":"49b86c887b8074a7e83dd62a5efc6cd9a1207ade87cb11c640fa095e16679079"} Oct 31 13:44:59.544169 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.544137 5074 generic.go:356] "Generic (PLEG): container finished" podID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerID="4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186" exitCode=0 Oct 31 13:44:59.544311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.544180 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" Oct 31 13:44:59.544311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.544205 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" event={"ID":"8ff51ea1-b218-4ca6-ac95-d81bdabe324f","Type":"ContainerDied","Data":"4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186"} Oct 31 13:44:59.544311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.544251 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr" event={"ID":"8ff51ea1-b218-4ca6-ac95-d81bdabe324f","Type":"ContainerDied","Data":"627aaf5cb2401074197d4aeca6a9ec6f1ea3c4f7a0d4873d4c9a865468a8dda3"} Oct 31 13:44:59.544311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.544273 5074 scope.go:122] "RemoveContainer" containerID="4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186" Oct 31 13:44:59.556203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.556179 5074 scope.go:122] "RemoveContainer" containerID="af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51" Oct 31 13:44:59.566963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.566942 5074 scope.go:122] "RemoveContainer" containerID="4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186" Oct 31 13:44:59.567298 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:59.567264 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186\": container with ID starting with 4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186 not found: ID does not exist" containerID="4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186" Oct 31 13:44:59.567435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.567309 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186"} err="failed to get container status \"4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186\": rpc error: code = NotFound desc = could not find container \"4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186\": container with ID starting with 4488683ef6f8ea82699518768db416b4447d2ddf1b183ab3338a956147053186 not found: ID does not exist" Oct 31 13:44:59.567435 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.567335 5074 scope.go:122] "RemoveContainer" containerID="af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51" Oct 31 13:44:59.567720 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:44:59.567686 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51\": container with ID starting with af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51 not found: ID does not exist" containerID="af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51" Oct 31 13:44:59.567848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.567729 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51"} err="failed to get container status \"af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51\": rpc error: code = NotFound desc = could not find container \"af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51\": container with ID starting with af1d316a742173fe62be3c93b15b1789fa22be9a33fa27afb274ca89224cef51 not found: ID does not exist" Oct 31 13:44:59.591223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.591192 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr"] Oct 31 13:44:59.597091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.597062 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5452fc-openshift-default-6db8c8bb5-twwdr"] Oct 31 13:44:59.799296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.799182 5074 patch_prober.go:29] interesting pod/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:44:59.799296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.799249 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:44:59.938309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:44:59.938270 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" path="/var/lib/kubelet/pods/8ff51ea1-b218-4ca6-ac95-d81bdabe324f/volumes" Oct 31 13:45:00.166007 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.165925 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2"] Oct 31 13:45:00.166426 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166411 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" Oct 31 13:45:00.166471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166428 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" Oct 31 13:45:00.166471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166445 5074 cpu_manager.go:414] "containerMap: removing container" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" Oct 31 13:45:00.166471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166450 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" Oct 31 13:45:00.166471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166462 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b7c98812-e834-4f7e-91f4-41a22574c8e4" containerName="revoke-keys" Oct 31 13:45:00.166471 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166468 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c98812-e834-4f7e-91f4-41a22574c8e4" containerName="revoke-keys" Oct 31 13:45:00.166731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166562 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" Oct 31 13:45:00.166731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166574 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff51ea1-b218-4ca6-ac95-d81bdabe324f" containerName="istio-proxy" Oct 31 13:45:00.166731 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.166584 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c98812-e834-4f7e-91f4-41a22574c8e4" containerName="revoke-keys" Oct 31 13:45:00.171702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.171673 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.173958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.173933 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc"] Oct 31 13:45:00.177758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.177738 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.210940 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.210916 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2"] Oct 31 13:45:00.215766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.215743 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj"] Oct 31 13:45:00.220079 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.220051 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.220182 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.220061 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc"] Oct 31 13:45:00.273408 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.273372 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj"] Oct 31 13:45:00.285783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.285743 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.285943 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.285880 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bda705-a0e5-4214-b53d-557561d0a67f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.286023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.285936 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ml2g\" (UniqueName: \"kubernetes.io/projected/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-kube-api-access-9ml2g\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.286091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.286068 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc96ebae-a48c-4fff-8192-a013c6eaaead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.286153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.286135 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fk5\" (UniqueName: \"kubernetes.io/projected/fc96ebae-a48c-4fff-8192-a013c6eaaead-kube-api-access-n4fk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.286279 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.286221 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqgv\" (UniqueName: \"kubernetes.io/projected/37bda705-a0e5-4214-b53d-557561d0a67f-kube-api-access-7vqgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.294882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.294853 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh"] Oct 31 13:45:00.299938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.299913 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.318769 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.318734 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh"] Oct 31 13:45:00.387198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387164 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.387381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387224 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bda705-a0e5-4214-b53d-557561d0a67f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.387381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387254 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ml2g\" (UniqueName: \"kubernetes.io/projected/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-kube-api-access-9ml2g\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.387381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387318 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc96ebae-a48c-4fff-8192-a013c6eaaead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.387381 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387354 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fk5\" (UniqueName: \"kubernetes.io/projected/fc96ebae-a48c-4fff-8192-a013c6eaaead-kube-api-access-n4fk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.387582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387386 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwbx\" (UniqueName: \"kubernetes.io/projected/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-kube-api-access-kmwbx\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.387582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387416 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.387582 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.387445 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqgv\" (UniqueName: \"kubernetes.io/projected/37bda705-a0e5-4214-b53d-557561d0a67f-kube-api-access-7vqgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.388611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.388403 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.388611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.388566 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc96ebae-a48c-4fff-8192-a013c6eaaead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.389787 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.389762 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bda705-a0e5-4214-b53d-557561d0a67f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.407258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.407227 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqgv\" (UniqueName: \"kubernetes.io/projected/37bda705-a0e5-4214-b53d-557561d0a67f-kube-api-access-7vqgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.408619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.408577 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ml2g\" (UniqueName: \"kubernetes.io/projected/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-kube-api-access-9ml2g\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.408619 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.408596 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fk5\" (UniqueName: \"kubernetes.io/projected/fc96ebae-a48c-4fff-8192-a013c6eaaead-kube-api-access-n4fk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.486766 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.486727 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:00.488214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.488173 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwbx\" (UniqueName: \"kubernetes.io/projected/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-kube-api-access-kmwbx\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.488374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.488236 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.489153 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.489127 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.493161 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.493138 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:00.501398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.501373 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwbx\" (UniqueName: \"kubernetes.io/projected/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-kube-api-access-kmwbx\") pod \"maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.536062 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.534915 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:00.551081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.551033 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 13:45:00.588353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.588312 5074 generic.go:356] "Generic (PLEG): container finished" podID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerID="e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7" exitCode=0 Oct 31 13:45:00.589078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589047 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-podinfo\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589110 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-token\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-token\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589255 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589221 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istiod-ca-cert\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589311 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589284 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-kube-api-access-r745d\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-kube-api-access-r745d\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589337 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-certs\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589421 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-credential-socket\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589466 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-socket\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589550 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-envoy\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.589669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.589646 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-data\") pod \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\" (UID: \"d2d18089-6846-4b29-bf2d-8b0d1e024b17\") " Oct 31 13:45:00.593598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.593575 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-credential-socket" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.595456 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.595426 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-certs" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.596411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.596387 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.596587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.596572 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.597132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.597056 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-kube-api-access-r745d" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "kube-api-access-r745d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:00.597230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.597220 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" Oct 31 13:45:00.597353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.597327 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-socket" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.595560 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-data" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.600840 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-podinfo" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.597422 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" event={"ID":"d2d18089-6846-4b29-bf2d-8b0d1e024b17","Type":"ContainerDied","Data":"e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7"} Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.601282 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7" event={"ID":"d2d18089-6846-4b29-bf2d-8b0d1e024b17","Type":"ContainerDied","Data":"06b96559c610dfc272b2cdef7c2785aeced60abc3b33a4c7ac816b5bb38a3f14"} Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.601725 5074 scope.go:122] "RemoveContainer" containerID="e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7" Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.602034 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istiod-ca-cert" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.604518 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-token" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:00.609368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.608101 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-envoy" pod "d2d18089-6846-4b29-bf2d-8b0d1e024b17" (UID: "d2d18089-6846-4b29-bf2d-8b0d1e024b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:00.627895 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.627617 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:00.640777 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.640458 5074 scope.go:122] "RemoveContainer" containerID="e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7" Oct 31 13:45:00.641416 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:00.641292 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7\": container with ID starting with e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7 not found: ID does not exist" containerID="e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7" Oct 31 13:45:00.641416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.641342 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7"} err="failed to get container status \"e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7\": rpc error: code = NotFound desc = could not find container \"e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7\": container with ID starting with e1533e5f311d6dd47342cab67841b8cfe6ff907ce133cb8fe11ca891df8731d7 not found: ID does not exist" Oct 31 13:45:00.698354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.698243 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.698354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.698273 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.698354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.698285 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.698354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.698294 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.698354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.698304 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d18089-6846-4b29-bf2d-8b0d1e024b17-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.698354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.698313 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r745d\" (UniqueName: \"kubernetes.io/projected/d2d18089-6846-4b29-bf2d-8b0d1e024b17-kube-api-access-r745d\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.698354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.698323 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d18089-6846-4b29-bf2d-8b0d1e024b17-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.819863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.819834 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:45:00.868317 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:45:00.868285 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a55780_f5ee_4565_82ed_b4eaaa2b4ad9.slice/crio-b36f220a3fdeb0723b0e53656e7dd02ab23c6fd70913169307f0b227ddd1a144 WatchSource:0}: Error finding container b36f220a3fdeb0723b0e53656e7dd02ab23c6fd70913169307f0b227ddd1a144: Status 404 returned error can't find the container with id b36f220a3fdeb0723b0e53656e7dd02ab23c6fd70913169307f0b227ddd1a144 Oct 31 13:45:00.869888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.869816 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc"] Oct 31 13:45:00.894898 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.894856 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2"] Oct 31 13:45:00.905002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.904594 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7842b9dc-893b-4c00-90b2-8283e5b763b3-kube-api-access-n9h2s\" (UniqueName: \"kubernetes.io/projected/7842b9dc-893b-4c00-90b2-8283e5b763b3-kube-api-access-n9h2s\") pod \"7842b9dc-893b-4c00-90b2-8283e5b763b3\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " Oct 31 13:45:00.905002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.904692 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-openshift-ca\") pod \"7842b9dc-893b-4c00-90b2-8283e5b763b3\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " Oct 31 13:45:00.905002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.904840 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-xks-ca\") pod \"7842b9dc-893b-4c00-90b2-8283e5b763b3\" (UID: \"7842b9dc-893b-4c00-90b2-8283e5b763b3\") " Oct 31 13:45:00.905242 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.905007 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj"] Oct 31 13:45:00.905995 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.905883 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-openshift-ca" pod "7842b9dc-893b-4c00-90b2-8283e5b763b3" (UID: "7842b9dc-893b-4c00-90b2-8283e5b763b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:00.908029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.907989 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7842b9dc-893b-4c00-90b2-8283e5b763b3-kube-api-access-n9h2s" pod "7842b9dc-893b-4c00-90b2-8283e5b763b3" (UID: "7842b9dc-893b-4c00-90b2-8283e5b763b3"). InnerVolumeSpecName "kube-api-access-n9h2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:00.909288 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:45:00.909262 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bda705_a0e5_4214_b53d_557561d0a67f.slice/crio-ad11a8787477cdd3b5f0763b4842c529c82a4a1d25c3893f009ee1d8b410afe6 WatchSource:0}: Error finding container ad11a8787477cdd3b5f0763b4842c529c82a4a1d25c3893f009ee1d8b410afe6: Status 404 returned error can't find the container with id ad11a8787477cdd3b5f0763b4842c529c82a4a1d25c3893f009ee1d8b410afe6 Oct 31 13:45:00.909628 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.909594 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-xks-ca" pod "7842b9dc-893b-4c00-90b2-8283e5b763b3" (UID: "7842b9dc-893b-4c00-90b2-8283e5b763b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:00.942427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.942398 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7"] Oct 31 13:45:00.952737 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.952712 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3dd1e5-openshift-default-59458dd6d5-7znw7"] Oct 31 13:45:00.966571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:00.966547 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh"] Oct 31 13:45:01.006725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.006591 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9h2s\" (UniqueName: \"kubernetes.io/projected/7842b9dc-893b-4c00-90b2-8283e5b763b3-kube-api-access-n9h2s\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:01.006725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.006647 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:01.006725 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.006667 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7842b9dc-893b-4c00-90b2-8283e5b763b3-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:01.597670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.597523 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" event={"ID":"37bda705-a0e5-4214-b53d-557561d0a67f","Type":"ContainerStarted","Data":"aa5ba5b0a737942c3975e1f3e5f035a73ac924f6a49684cbe789c9a90377edfa"} Oct 31 13:45:01.597670 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.597590 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" event={"ID":"37bda705-a0e5-4214-b53d-557561d0a67f","Type":"ContainerStarted","Data":"ad11a8787477cdd3b5f0763b4842c529c82a4a1d25c3893f009ee1d8b410afe6"} Oct 31 13:45:01.599124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.599101 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" Oct 31 13:45:01.599124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.599113 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59" event={"ID":"7842b9dc-893b-4c00-90b2-8283e5b763b3","Type":"ContainerDied","Data":"d8f65ca387a8ea799de62d35e027d981503d0cc4f7f45774698d1a71a4656bd7"} Oct 31 13:45:01.599315 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.599154 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f65ca387a8ea799de62d35e027d981503d0cc4f7f45774698d1a71a4656bd7" Oct 31 13:45:01.601767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.601108 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerStarted","Data":"d3b32a4e7dcad132fc36982fe8c56ee49f4dce5fb215672c60c819c50946ab95"} Oct 31 13:45:01.601767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.601156 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerStarted","Data":"840a9ce847e95a07fa9acf46c2d0652ff4322ce070cc2ba01932962b2ac1c4d1"} Oct 31 13:45:01.603382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.603358 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerStarted","Data":"2277473833f9070bd94433e2b502cd8f50876683247face5e827660a25a34fb4"} Oct 31 13:45:01.603515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.603401 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerStarted","Data":"b36f220a3fdeb0723b0e53656e7dd02ab23c6fd70913169307f0b227ddd1a144"} Oct 31 13:45:01.606058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.606033 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerStarted","Data":"aedcf872cb5f582428329558a12181a20ba0bc9522bfea47dd97d04a9198ea71"} Oct 31 13:45:01.606163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.606071 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerStarted","Data":"ea5ee748d894a2c969318ce4782a84021650719fe9421f53bb7c234fee672b7b"} Oct 31 13:45:01.619869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.619821 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" podStartSLOduration=1.619808938 podStartE2EDuration="1.619808938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:01.615373106 +0000 UTC m=+4395.793091091" watchObservedRunningTime="2026-10-31 13:45:01.619808938 +0000 UTC m=+4395.797526922" Oct 31 13:45:01.640025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.639978 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" podStartSLOduration=1.6399665479999999 podStartE2EDuration="1.639966548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:01.636402371 +0000 UTC m=+4395.814120356" watchObservedRunningTime="2026-10-31 13:45:01.639966548 +0000 UTC m=+4395.817684532" Oct 31 13:45:01.661626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.661563 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" podStartSLOduration=1.6615468469999999 podStartE2EDuration="1.661546847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:01.660552456 +0000 UTC m=+4395.838270442" watchObservedRunningTime="2026-10-31 13:45:01.661546847 +0000 UTC m=+4395.839264831" Oct 31 13:45:01.689528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.689459 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" podStartSLOduration=1.689439957 podStartE2EDuration="1.689439957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:01.685139251 +0000 UTC m=+4395.862857235" watchObservedRunningTime="2026-10-31 13:45:01.689439957 +0000 UTC m=+4395.867157976" Oct 31 13:45:01.940546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:01.940512 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" path="/var/lib/kubelet/pods/d2d18089-6846-4b29-bf2d-8b0d1e024b17/volumes" Oct 31 13:45:02.226550 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.226517 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" podUID="d9ab647a-b861-4530-8d48-f9a05832887b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:02.615018 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.614980 5074 generic.go:356] "Generic (PLEG): container finished" podID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerID="d3b32a4e7dcad132fc36982fe8c56ee49f4dce5fb215672c60c819c50946ab95" exitCode=60 Oct 31 13:45:02.615262 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.615084 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerDied","Data":"d3b32a4e7dcad132fc36982fe8c56ee49f4dce5fb215672c60c819c50946ab95"} Oct 31 13:45:02.615460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.615436 5074 scope.go:122] "RemoveContainer" containerID="d3b32a4e7dcad132fc36982fe8c56ee49f4dce5fb215672c60c819c50946ab95" Oct 31 13:45:02.617548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.617523 5074 generic.go:356] "Generic (PLEG): container finished" podID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerID="2277473833f9070bd94433e2b502cd8f50876683247face5e827660a25a34fb4" exitCode=60 Oct 31 13:45:02.617659 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.617620 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerDied","Data":"2277473833f9070bd94433e2b502cd8f50876683247face5e827660a25a34fb4"} Oct 31 13:45:02.617945 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.617926 5074 scope.go:122] "RemoveContainer" containerID="2277473833f9070bd94433e2b502cd8f50876683247face5e827660a25a34fb4" Oct 31 13:45:02.619541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.619518 5074 generic.go:356] "Generic (PLEG): container finished" podID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerID="aedcf872cb5f582428329558a12181a20ba0bc9522bfea47dd97d04a9198ea71" exitCode=60 Oct 31 13:45:02.619668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.619545 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerDied","Data":"aedcf872cb5f582428329558a12181a20ba0bc9522bfea47dd97d04a9198ea71"} Oct 31 13:45:02.619852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.619834 5074 scope.go:122] "RemoveContainer" containerID="aedcf872cb5f582428329558a12181a20ba0bc9522bfea47dd97d04a9198ea71" Oct 31 13:45:02.621325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.621306 5074 generic.go:356] "Generic (PLEG): container finished" podID="37bda705-a0e5-4214-b53d-557561d0a67f" containerID="aa5ba5b0a737942c3975e1f3e5f035a73ac924f6a49684cbe789c9a90377edfa" exitCode=60 Oct 31 13:45:02.621414 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.621331 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" event={"ID":"37bda705-a0e5-4214-b53d-557561d0a67f","Type":"ContainerDied","Data":"aa5ba5b0a737942c3975e1f3e5f035a73ac924f6a49684cbe789c9a90377edfa"} Oct 31 13:45:02.621646 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:02.621618 5074 scope.go:122] "RemoveContainer" containerID="aa5ba5b0a737942c3975e1f3e5f035a73ac924f6a49684cbe789c9a90377edfa" Oct 31 13:45:03.629999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:03.629954 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerStarted","Data":"3d8a4d4f0f59a8b986fbdbcdd725040dadc1e20b44ae2a7e5bc864e1a894af17"} Oct 31 13:45:03.632107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:03.632081 5074 generic.go:356] "Generic (PLEG): container finished" podID="37bda705-a0e5-4214-b53d-557561d0a67f" containerID="c8e170eea1c8bf865ab036a1be4ed458e074e7279b294d0d2ecd781f2a701808" exitCode=60 Oct 31 13:45:03.632263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:03.632174 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" event={"ID":"37bda705-a0e5-4214-b53d-557561d0a67f","Type":"ContainerDied","Data":"c8e170eea1c8bf865ab036a1be4ed458e074e7279b294d0d2ecd781f2a701808"} Oct 31 13:45:03.632263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:03.632235 5074 scope.go:122] "RemoveContainer" containerID="aa5ba5b0a737942c3975e1f3e5f035a73ac924f6a49684cbe789c9a90377edfa" Oct 31 13:45:03.632438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:03.632423 5074 scope.go:122] "RemoveContainer" containerID="c8e170eea1c8bf865ab036a1be4ed458e074e7279b294d0d2ecd781f2a701808" Oct 31 13:45:03.632779 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:03.632753 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj_odh-ai-gateway-infra(37bda705-a0e5-4214-b53d-557561d0a67f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" Oct 31 13:45:03.634591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:03.634566 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerStarted","Data":"eea28a63d3ff2ed23fe51bbad149479db45bc32657950dfb790a9f8a65374520"} Oct 31 13:45:03.636566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:03.636542 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerStarted","Data":"3fe63b38ee6caf631b9024368a4a43d58598d402f1fac9e8624dd0591a98102f"} Oct 31 13:45:04.647029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.646992 5074 generic.go:356] "Generic (PLEG): container finished" podID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerID="3d8a4d4f0f59a8b986fbdbcdd725040dadc1e20b44ae2a7e5bc864e1a894af17" exitCode=60 Oct 31 13:45:04.647519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.647078 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerDied","Data":"3d8a4d4f0f59a8b986fbdbcdd725040dadc1e20b44ae2a7e5bc864e1a894af17"} Oct 31 13:45:04.647519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.647153 5074 scope.go:122] "RemoveContainer" containerID="aedcf872cb5f582428329558a12181a20ba0bc9522bfea47dd97d04a9198ea71" Oct 31 13:45:04.647519 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.647378 5074 scope.go:122] "RemoveContainer" containerID="3d8a4d4f0f59a8b986fbdbcdd725040dadc1e20b44ae2a7e5bc864e1a894af17" Oct 31 13:45:04.647728 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:04.647706 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh_odh-ai-gateway-infra(fa537dee-f5b5-4c47-932f-3c9ee78d9c8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" Oct 31 13:45:04.649240 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.649219 5074 scope.go:122] "RemoveContainer" containerID="c8e170eea1c8bf865ab036a1be4ed458e074e7279b294d0d2ecd781f2a701808" Oct 31 13:45:04.649513 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:04.649493 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj_odh-ai-gateway-infra(37bda705-a0e5-4214-b53d-557561d0a67f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" Oct 31 13:45:04.650906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.650885 5074 generic.go:356] "Generic (PLEG): container finished" podID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerID="eea28a63d3ff2ed23fe51bbad149479db45bc32657950dfb790a9f8a65374520" exitCode=60 Oct 31 13:45:04.650993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.650977 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerDied","Data":"eea28a63d3ff2ed23fe51bbad149479db45bc32657950dfb790a9f8a65374520"} Oct 31 13:45:04.651236 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.651213 5074 scope.go:122] "RemoveContainer" containerID="eea28a63d3ff2ed23fe51bbad149479db45bc32657950dfb790a9f8a65374520" Oct 31 13:45:04.651509 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:04.651478 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2_odh-ai-gateway-infra(fc96ebae-a48c-4fff-8192-a013c6eaaead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" Oct 31 13:45:04.653006 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.652984 5074 generic.go:356] "Generic (PLEG): container finished" podID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerID="3fe63b38ee6caf631b9024368a4a43d58598d402f1fac9e8624dd0591a98102f" exitCode=60 Oct 31 13:45:04.653087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.653050 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerDied","Data":"3fe63b38ee6caf631b9024368a4a43d58598d402f1fac9e8624dd0591a98102f"} Oct 31 13:45:04.653253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.653240 5074 scope.go:122] "RemoveContainer" containerID="3fe63b38ee6caf631b9024368a4a43d58598d402f1fac9e8624dd0591a98102f" Oct 31 13:45:04.653417 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:04.653401 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc_odh-ai-gateway-infra(d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" Oct 31 13:45:04.660461 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.660429 5074 scope.go:122] "RemoveContainer" containerID="d3b32a4e7dcad132fc36982fe8c56ee49f4dce5fb215672c60c819c50946ab95" Oct 31 13:45:04.675816 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:04.675786 5074 scope.go:122] "RemoveContainer" containerID="2277473833f9070bd94433e2b502cd8f50876683247face5e827660a25a34fb4" Oct 31 13:45:05.661709 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:05.661674 5074 scope.go:122] "RemoveContainer" containerID="3d8a4d4f0f59a8b986fbdbcdd725040dadc1e20b44ae2a7e5bc864e1a894af17" Oct 31 13:45:05.662123 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:05.661939 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh_odh-ai-gateway-infra(fa537dee-f5b5-4c47-932f-3c9ee78d9c8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" Oct 31 13:45:05.663270 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:05.663249 5074 scope.go:122] "RemoveContainer" containerID="eea28a63d3ff2ed23fe51bbad149479db45bc32657950dfb790a9f8a65374520" Oct 31 13:45:05.663477 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:05.663458 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2_odh-ai-gateway-infra(fc96ebae-a48c-4fff-8192-a013c6eaaead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" Oct 31 13:45:05.664685 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:05.664667 5074 scope.go:122] "RemoveContainer" containerID="3fe63b38ee6caf631b9024368a4a43d58598d402f1fac9e8624dd0591a98102f" Oct 31 13:45:05.664913 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:05.664894 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc_odh-ai-gateway-infra(d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" Oct 31 13:45:07.598324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.598287 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:45:07.684705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.684622 5074 generic.go:356] "Generic (PLEG): container finished" podID="d9ab647a-b861-4530-8d48-f9a05832887b" containerID="60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a" exitCode=0 Oct 31 13:45:07.684929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.684738 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" event={"ID":"d9ab647a-b861-4530-8d48-f9a05832887b","Type":"ContainerDied","Data":"60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a"} Oct 31 13:45:07.684929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.684796 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" event={"ID":"d9ab647a-b861-4530-8d48-f9a05832887b","Type":"ContainerDied","Data":"65487be109e5726c80fd22927cf075bed247bd145488c5c52be676952ae625b1"} Oct 31 13:45:07.684929 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.684829 5074 scope.go:122] "RemoveContainer" containerID="60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a" Oct 31 13:45:07.685127 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.684947 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr" Oct 31 13:45:07.707168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.707133 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ab647a-b861-4530-8d48-f9a05832887b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab647a-b861-4530-8d48-f9a05832887b-plugins-config-volume\") pod \"d9ab647a-b861-4530-8d48-f9a05832887b\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " Oct 31 13:45:07.707557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.707536 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ab647a-b861-4530-8d48-f9a05832887b-kube-api-access-4724m\" (UniqueName: \"kubernetes.io/projected/d9ab647a-b861-4530-8d48-f9a05832887b-kube-api-access-4724m\") pod \"d9ab647a-b861-4530-8d48-f9a05832887b\" (UID: \"d9ab647a-b861-4530-8d48-f9a05832887b\") " Oct 31 13:45:07.708862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.708837 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ab647a-b861-4530-8d48-f9a05832887b-plugins-config-volume" pod "d9ab647a-b861-4530-8d48-f9a05832887b" (UID: "d9ab647a-b861-4530-8d48-f9a05832887b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:07.724566 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.720216 5074 scope.go:122] "RemoveContainer" containerID="60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a" Oct 31 13:45:07.724782 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:07.724745 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a\": container with ID starting with 60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a not found: ID does not exist" containerID="60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a" Oct 31 13:45:07.724897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.724795 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a"} err="failed to get container status \"60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a\": rpc error: code = NotFound desc = could not find container \"60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a\": container with ID starting with 60dbac32e21f9dfad10e5a5e3c09f202058caea0b6d17cc35ab274a0758a408a not found: ID does not exist" Oct 31 13:45:07.724897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.724835 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ab647a-b861-4530-8d48-f9a05832887b-kube-api-access-4724m" pod "d9ab647a-b861-4530-8d48-f9a05832887b" (UID: "d9ab647a-b861-4530-8d48-f9a05832887b"). InnerVolumeSpecName "kube-api-access-4724m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:07.809801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.809683 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4724m\" (UniqueName: \"kubernetes.io/projected/d9ab647a-b861-4530-8d48-f9a05832887b-kube-api-access-4724m\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:07.809801 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.809717 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab647a-b861-4530-8d48-f9a05832887b-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:07.970384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.970348 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4"] Oct 31 13:45:07.970894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.970878 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7842b9dc-893b-4c00-90b2-8283e5b763b3" containerName="revoke-keys" Oct 31 13:45:07.970969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.970894 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7842b9dc-893b-4c00-90b2-8283e5b763b3" containerName="revoke-keys" Oct 31 13:45:07.970969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.970908 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d9ab647a-b861-4530-8d48-f9a05832887b" containerName="payload-processing" Oct 31 13:45:07.970969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.970914 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ab647a-b861-4530-8d48-f9a05832887b" containerName="payload-processing" Oct 31 13:45:07.970969 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.970963 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" Oct 31 13:45:07.971165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.970972 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" Oct 31 13:45:07.971165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.971065 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d18089-6846-4b29-bf2d-8b0d1e024b17" containerName="istio-proxy" Oct 31 13:45:07.971165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.971072 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7842b9dc-893b-4c00-90b2-8283e5b763b3" containerName="revoke-keys" Oct 31 13:45:07.971165 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.971080 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ab647a-b861-4530-8d48-f9a05832887b" containerName="payload-processing" Oct 31 13:45:07.975982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.975963 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:07.982894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.982868 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:45:07.990181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:07.990147 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4"] Oct 31 13:45:08.035918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.035879 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr"] Oct 31 13:45:08.051235 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.051205 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5452fc-6b57bc4f5-bl5pr"] Oct 31 13:45:08.113076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.112985 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a588df7-2573-462a-b898-637c548384a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.113249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.113117 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjmt\" (UniqueName: \"kubernetes.io/projected/7a588df7-2573-462a-b898-637c548384a1-kube-api-access-nnjmt\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.113249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.113180 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a588df7-2573-462a-b898-637c548384a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.214518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.214477 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjmt\" (UniqueName: \"kubernetes.io/projected/7a588df7-2573-462a-b898-637c548384a1-kube-api-access-nnjmt\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.214773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.214545 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a588df7-2573-462a-b898-637c548384a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.214773 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.214627 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a588df7-2573-462a-b898-637c548384a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.215518 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.215489 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a588df7-2573-462a-b898-637c548384a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.217012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.216986 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a588df7-2573-462a-b898-637c548384a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.227992 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.227956 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjmt\" (UniqueName: \"kubernetes.io/projected/7a588df7-2573-462a-b898-637c548384a1-kube-api-access-nnjmt\") pod \"maas-api-revoke-sub-adbf77dd-4kll4\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.294304 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.294273 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:08.583537 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.583504 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4"] Oct 31 13:45:08.584928 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:45:08.584891 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a588df7_2573_462a_b898_637c548384a1.slice/crio-1b1bff5efb284bc47afcf5235f8f70a5f411e63a75b674e09d77a7467ca6bc95 WatchSource:0}: Error finding container 1b1bff5efb284bc47afcf5235f8f70a5f411e63a75b674e09d77a7467ca6bc95: Status 404 returned error can't find the container with id 1b1bff5efb284bc47afcf5235f8f70a5f411e63a75b674e09d77a7467ca6bc95 Oct 31 13:45:08.695793 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:08.695749 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" event={"ID":"7a588df7-2573-462a-b898-637c548384a1","Type":"ContainerStarted","Data":"1b1bff5efb284bc47afcf5235f8f70a5f411e63a75b674e09d77a7467ca6bc95"} Oct 31 13:45:09.704982 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:09.704943 5074 generic.go:356] "Generic (PLEG): container finished" podID="7a588df7-2573-462a-b898-637c548384a1" containerID="0d9dd7bc3b3e22c43d968a5439e38985ed5676f9d6045fee207af4d3877f16b9" exitCode=0 Oct 31 13:45:09.705394 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:09.704992 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" event={"ID":"7a588df7-2573-462a-b898-637c548384a1","Type":"ContainerDied","Data":"0d9dd7bc3b3e22c43d968a5439e38985ed5676f9d6045fee207af4d3877f16b9"} Oct 31 13:45:09.938600 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:09.938513 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ab647a-b861-4530-8d48-f9a05832887b" path="/var/lib/kubelet/pods/d9ab647a-b861-4530-8d48-f9a05832887b/volumes" Oct 31 13:45:10.876746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:10.876710 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:10.946760 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:10.946720 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a588df7-2573-462a-b898-637c548384a1-kube-api-access-nnjmt\" (UniqueName: \"kubernetes.io/projected/7a588df7-2573-462a-b898-637c548384a1-kube-api-access-nnjmt\") pod \"7a588df7-2573-462a-b898-637c548384a1\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " Oct 31 13:45:10.947012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:10.946863 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a588df7-2573-462a-b898-637c548384a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a588df7-2573-462a-b898-637c548384a1-maas-api-xks-ca\") pod \"7a588df7-2573-462a-b898-637c548384a1\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " Oct 31 13:45:10.947012 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:10.946917 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a588df7-2573-462a-b898-637c548384a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a588df7-2573-462a-b898-637c548384a1-maas-api-openshift-ca\") pod \"7a588df7-2573-462a-b898-637c548384a1\" (UID: \"7a588df7-2573-462a-b898-637c548384a1\") " Oct 31 13:45:10.947327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:10.947293 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a588df7-2573-462a-b898-637c548384a1-maas-api-openshift-ca" pod "7a588df7-2573-462a-b898-637c548384a1" (UID: "7a588df7-2573-462a-b898-637c548384a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:10.948970 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:10.948943 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a588df7-2573-462a-b898-637c548384a1-kube-api-access-nnjmt" pod "7a588df7-2573-462a-b898-637c548384a1" (UID: "7a588df7-2573-462a-b898-637c548384a1"). InnerVolumeSpecName "kube-api-access-nnjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:10.949054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:10.948988 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a588df7-2573-462a-b898-637c548384a1-maas-api-xks-ca" pod "7a588df7-2573-462a-b898-637c548384a1" (UID: "7a588df7-2573-462a-b898-637c548384a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:11.048788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:11.048754 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a588df7-2573-462a-b898-637c548384a1-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:11.048788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:11.048785 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a588df7-2573-462a-b898-637c548384a1-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:11.048985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:11.048798 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjmt\" (UniqueName: \"kubernetes.io/projected/7a588df7-2573-462a-b898-637c548384a1-kube-api-access-nnjmt\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:11.719259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:11.719223 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" event={"ID":"7a588df7-2573-462a-b898-637c548384a1","Type":"ContainerDied","Data":"1b1bff5efb284bc47afcf5235f8f70a5f411e63a75b674e09d77a7467ca6bc95"} Oct 31 13:45:11.719259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:11.719245 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4" Oct 31 13:45:11.719475 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:11.719273 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b1bff5efb284bc47afcf5235f8f70a5f411e63a75b674e09d77a7467ca6bc95" Oct 31 13:45:13.943721 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:13.943682 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" containerID="cri-o://b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b" gracePeriod=45 Oct 31 13:45:13.967036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:13.966997 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" containerID="cri-o://d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2" gracePeriod=45 Oct 31 13:45:13.978922 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:13.978880 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" containerID="cri-o://5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58" gracePeriod=45 Oct 31 13:45:13.992379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:13.992348 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" containerID="cri-o://8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6" gracePeriod=45 Oct 31 13:45:14.289326 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.289028 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 13:45:14.337743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.337463 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 13:45:14.345886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.345857 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 13:45:14.380609 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.380575 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 13:45:14.394075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.393906 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296a60e8-1efa-44da-bed6-842ccefdd93d-kube-api-access-8kjfr\" (UniqueName: \"kubernetes.io/projected/296a60e8-1efa-44da-bed6-842ccefdd93d-kube-api-access-8kjfr\") pod \"296a60e8-1efa-44da-bed6-842ccefdd93d\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " Oct 31 13:45:14.394075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.394042 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-dshm\") pod \"296a60e8-1efa-44da-bed6-842ccefdd93d\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " Oct 31 13:45:14.394339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.394116 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-tmp-dir\") pod \"296a60e8-1efa-44da-bed6-842ccefdd93d\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " Oct 31 13:45:14.394339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.394146 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-model-cache\") pod \"296a60e8-1efa-44da-bed6-842ccefdd93d\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " Oct 31 13:45:14.394509 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.394245 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-home\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-home\") pod \"296a60e8-1efa-44da-bed6-842ccefdd93d\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " Oct 31 13:45:14.394598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.394531 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/296a60e8-1efa-44da-bed6-842ccefdd93d-tls-certs\" (UniqueName: \"kubernetes.io/secret/296a60e8-1efa-44da-bed6-842ccefdd93d-tls-certs\") pod \"296a60e8-1efa-44da-bed6-842ccefdd93d\" (UID: \"296a60e8-1efa-44da-bed6-842ccefdd93d\") " Oct 31 13:45:14.408134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.399991 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-home" pod "296a60e8-1efa-44da-bed6-842ccefdd93d" (UID: "296a60e8-1efa-44da-bed6-842ccefdd93d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.408134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.401462 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296a60e8-1efa-44da-bed6-842ccefdd93d-tls-certs" pod "296a60e8-1efa-44da-bed6-842ccefdd93d" (UID: "296a60e8-1efa-44da-bed6-842ccefdd93d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:14.408134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.403678 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-dshm" pod "296a60e8-1efa-44da-bed6-842ccefdd93d" (UID: "296a60e8-1efa-44da-bed6-842ccefdd93d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.408134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.404760 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296a60e8-1efa-44da-bed6-842ccefdd93d-kube-api-access-8kjfr" pod "296a60e8-1efa-44da-bed6-842ccefdd93d" (UID: "296a60e8-1efa-44da-bed6-842ccefdd93d"). InnerVolumeSpecName "kube-api-access-8kjfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:14.408134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.405727 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-tmp-dir" pod "296a60e8-1efa-44da-bed6-842ccefdd93d" (UID: "296a60e8-1efa-44da-bed6-842ccefdd93d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.408134 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.407241 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-model-cache" pod "296a60e8-1efa-44da-bed6-842ccefdd93d" (UID: "296a60e8-1efa-44da-bed6-842ccefdd93d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.495877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.495794 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d94beb7c-137f-4c8c-b08e-cf95d981f725-tls-certs\" (UniqueName: \"kubernetes.io/secret/d94beb7c-137f-4c8c-b08e-cf95d981f725-tls-certs\") pod \"d94beb7c-137f-4c8c-b08e-cf95d981f725\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " Oct 31 13:45:14.495877 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.495844 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-tmp-dir\") pod \"d94beb7c-137f-4c8c-b08e-cf95d981f725\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " Oct 31 13:45:14.496120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.495885 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c03650-44c5-4c06-b8a3-940deaa36257-tls-certs\" (UniqueName: \"kubernetes.io/secret/66c03650-44c5-4c06-b8a3-940deaa36257-tls-certs\") pod \"66c03650-44c5-4c06-b8a3-940deaa36257\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " Oct 31 13:45:14.496120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.495931 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94beb7c-137f-4c8c-b08e-cf95d981f725-kube-api-access-p6xbh\" (UniqueName: \"kubernetes.io/projected/d94beb7c-137f-4c8c-b08e-cf95d981f725-kube-api-access-p6xbh\") pod \"d94beb7c-137f-4c8c-b08e-cf95d981f725\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " Oct 31 13:45:14.496120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.495982 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-dshm\") pod \"66c03650-44c5-4c06-b8a3-940deaa36257\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " Oct 31 13:45:14.496120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496059 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c03650-44c5-4c06-b8a3-940deaa36257-kube-api-access-plp4f\" (UniqueName: \"kubernetes.io/projected/66c03650-44c5-4c06-b8a3-940deaa36257-kube-api-access-plp4f\") pod \"66c03650-44c5-4c06-b8a3-940deaa36257\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " Oct 31 13:45:14.496120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496091 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-home\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-home\") pod \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " Oct 31 13:45:14.496120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496118 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-dshm\") pod \"d94beb7c-137f-4c8c-b08e-cf95d981f725\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496123 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-tmp-dir" pod "d94beb7c-137f-4c8c-b08e-cf95d981f725" (UID: "d94beb7c-137f-4c8c-b08e-cf95d981f725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496197 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tmp-dir\") pod \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496227 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-home\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-home\") pod \"d94beb7c-137f-4c8c-b08e-cf95d981f725\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496263 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-home\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-home\") pod \"66c03650-44c5-4c06-b8a3-940deaa36257\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496307 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-model-cache\") pod \"d94beb7c-137f-4c8c-b08e-cf95d981f725\" (UID: \"d94beb7c-137f-4c8c-b08e-cf95d981f725\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496338 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-model-cache\") pod \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496390 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-kube-api-access-5b4rz\" (UniqueName: \"kubernetes.io/projected/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-kube-api-access-5b4rz\") pod \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496417 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-tmp-dir\") pod \"66c03650-44c5-4c06-b8a3-940deaa36257\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " Oct 31 13:45:14.496460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496454 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-dshm\") pod \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " Oct 31 13:45:14.497021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496526 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-model-cache\") pod \"66c03650-44c5-4c06-b8a3-940deaa36257\" (UID: \"66c03650-44c5-4c06-b8a3-940deaa36257\") " Oct 31 13:45:14.497021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.496565 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tls-certs\") pod \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\" (UID: \"90a85d43-3cca-48ff-b8a3-f8de2beff5b1\") " Oct 31 13:45:14.497145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497060 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.497145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497081 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.497145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497095 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.497145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497110 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.497145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497123 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a60e8-1efa-44da-bed6-842ccefdd93d-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.497145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497137 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjfr\" (UniqueName: \"kubernetes.io/projected/296a60e8-1efa-44da-bed6-842ccefdd93d-kube-api-access-8kjfr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.497484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497150 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a60e8-1efa-44da-bed6-842ccefdd93d-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.497733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497707 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-model-cache" pod "90a85d43-3cca-48ff-b8a3-f8de2beff5b1" (UID: "90a85d43-3cca-48ff-b8a3-f8de2beff5b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.497821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497741 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-tmp-dir" pod "66c03650-44c5-4c06-b8a3-940deaa36257" (UID: "66c03650-44c5-4c06-b8a3-940deaa36257"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.497933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.497913 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tmp-dir" pod "90a85d43-3cca-48ff-b8a3-f8de2beff5b1" (UID: "90a85d43-3cca-48ff-b8a3-f8de2beff5b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.498128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.498106 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-home" pod "d94beb7c-137f-4c8c-b08e-cf95d981f725" (UID: "d94beb7c-137f-4c8c-b08e-cf95d981f725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.498345 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.498326 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-home" pod "66c03650-44c5-4c06-b8a3-940deaa36257" (UID: "66c03650-44c5-4c06-b8a3-940deaa36257"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.498523 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.498504 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-model-cache" pod "d94beb7c-137f-4c8c-b08e-cf95d981f725" (UID: "d94beb7c-137f-4c8c-b08e-cf95d981f725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.498702 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.498672 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94beb7c-137f-4c8c-b08e-cf95d981f725-kube-api-access-p6xbh" pod "d94beb7c-137f-4c8c-b08e-cf95d981f725" (UID: "d94beb7c-137f-4c8c-b08e-cf95d981f725"). InnerVolumeSpecName "kube-api-access-p6xbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:14.498927 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.498906 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-home" pod "90a85d43-3cca-48ff-b8a3-f8de2beff5b1" (UID: "90a85d43-3cca-48ff-b8a3-f8de2beff5b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.499099 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.499080 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-model-cache" pod "66c03650-44c5-4c06-b8a3-940deaa36257" (UID: "66c03650-44c5-4c06-b8a3-940deaa36257"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.499190 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.499077 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94beb7c-137f-4c8c-b08e-cf95d981f725-tls-certs" pod "d94beb7c-137f-4c8c-b08e-cf95d981f725" (UID: "d94beb7c-137f-4c8c-b08e-cf95d981f725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:14.499327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.499284 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c03650-44c5-4c06-b8a3-940deaa36257-kube-api-access-plp4f" pod "66c03650-44c5-4c06-b8a3-940deaa36257" (UID: "66c03650-44c5-4c06-b8a3-940deaa36257"). InnerVolumeSpecName "kube-api-access-plp4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:14.499327 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.499318 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c03650-44c5-4c06-b8a3-940deaa36257-tls-certs" pod "66c03650-44c5-4c06-b8a3-940deaa36257" (UID: "66c03650-44c5-4c06-b8a3-940deaa36257"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:14.500547 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.500498 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-kube-api-access-5b4rz" pod "90a85d43-3cca-48ff-b8a3-f8de2beff5b1" (UID: "90a85d43-3cca-48ff-b8a3-f8de2beff5b1"). InnerVolumeSpecName "kube-api-access-5b4rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:14.500718 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.500582 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-dshm" pod "d94beb7c-137f-4c8c-b08e-cf95d981f725" (UID: "d94beb7c-137f-4c8c-b08e-cf95d981f725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.500931 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.500905 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-dshm" pod "66c03650-44c5-4c06-b8a3-940deaa36257" (UID: "66c03650-44c5-4c06-b8a3-940deaa36257"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.501151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.501128 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tls-certs" pod "90a85d43-3cca-48ff-b8a3-f8de2beff5b1" (UID: "90a85d43-3cca-48ff-b8a3-f8de2beff5b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:14.501258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.501242 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-dshm" pod "90a85d43-3cca-48ff-b8a3-f8de2beff5b1" (UID: "90a85d43-3cca-48ff-b8a3-f8de2beff5b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:14.598533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598497 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c03650-44c5-4c06-b8a3-940deaa36257-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598531 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xbh\" (UniqueName: \"kubernetes.io/projected/d94beb7c-137f-4c8c-b08e-cf95d981f725-kube-api-access-p6xbh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598533 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598542 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598551 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plp4f\" (UniqueName: \"kubernetes.io/projected/66c03650-44c5-4c06-b8a3-940deaa36257-kube-api-access-plp4f\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598561 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598569 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598577 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598585 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598592 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598600 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94beb7c-137f-4c8c-b08e-cf95d981f725-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598609 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598618 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4rz\" (UniqueName: \"kubernetes.io/projected/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-kube-api-access-5b4rz\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598626 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598654 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598663 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c03650-44c5-4c06-b8a3-940deaa36257-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598670 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a85d43-3cca-48ff-b8a3-f8de2beff5b1-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.598897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.598679 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94beb7c-137f-4c8c-b08e-cf95d981f725-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:14.739274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.739241 5074 generic.go:356] "Generic (PLEG): container finished" podID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerID="d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2" exitCode=0 Oct 31 13:45:14.739447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.739283 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" Oct 31 13:45:14.739447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.739328 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" event={"ID":"296a60e8-1efa-44da-bed6-842ccefdd93d","Type":"ContainerDied","Data":"d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2"} Oct 31 13:45:14.739447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.739374 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" event={"ID":"296a60e8-1efa-44da-bed6-842ccefdd93d","Type":"ContainerDied","Data":"985a03abd44218fac271b9980c5b88dfa32b7938db53781ba8f02776c4d938d5"} Oct 31 13:45:14.739447 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.739400 5074 scope.go:122] "RemoveContainer" containerID="d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2" Oct 31 13:45:14.740726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.740625 5074 generic.go:356] "Generic (PLEG): container finished" podID="66c03650-44c5-4c06-b8a3-940deaa36257" containerID="b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b" exitCode=0 Oct 31 13:45:14.740726 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.740710 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" Oct 31 13:45:14.740868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.740735 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" event={"ID":"66c03650-44c5-4c06-b8a3-940deaa36257","Type":"ContainerDied","Data":"b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b"} Oct 31 13:45:14.740868 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.740776 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862" event={"ID":"66c03650-44c5-4c06-b8a3-940deaa36257","Type":"ContainerDied","Data":"a3aa34e4d4348a13402f5d9ece1b93a8b132ecc62cdca4e10d29c9798c3644e5"} Oct 31 13:45:14.742314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.742288 5074 generic.go:356] "Generic (PLEG): container finished" podID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerID="5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58" exitCode=0 Oct 31 13:45:14.742443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.742317 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" event={"ID":"90a85d43-3cca-48ff-b8a3-f8de2beff5b1","Type":"ContainerDied","Data":"5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58"} Oct 31 13:45:14.742443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.742332 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" Oct 31 13:45:14.742443 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.742350 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv" event={"ID":"90a85d43-3cca-48ff-b8a3-f8de2beff5b1","Type":"ContainerDied","Data":"fccc3fb4d61266d487dbd43a12a6c916032f4333630fef0261aa99eaefbe4f84"} Oct 31 13:45:14.744807 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.744784 5074 generic.go:356] "Generic (PLEG): container finished" podID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerID="8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6" exitCode=0 Oct 31 13:45:14.744923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.744822 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" Oct 31 13:45:14.744923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.744860 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" event={"ID":"d94beb7c-137f-4c8c-b08e-cf95d981f725","Type":"ContainerDied","Data":"8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6"} Oct 31 13:45:14.744923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.744891 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" event={"ID":"d94beb7c-137f-4c8c-b08e-cf95d981f725","Type":"ContainerDied","Data":"96ee085234ade5aedc816b94f55e4998ddfddb7025c6b05992d4b4a25c81dfa1"} Oct 31 13:45:14.758344 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.758309 5074 scope.go:122] "RemoveContainer" containerID="d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2" Oct 31 13:45:14.758594 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:14.758571 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2\": container with ID starting with d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2 not found: ID does not exist" containerID="d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2" Oct 31 13:45:14.758690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.758606 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2"} err="failed to get container status \"d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2\": rpc error: code = NotFound desc = could not find container \"d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2\": container with ID starting with d93dd2a2f7348acdd12448b3c7cce1d93a3f681a136a5c29ffd98b7820dc1ac2 not found: ID does not exist" Oct 31 13:45:14.758690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.758644 5074 scope.go:122] "RemoveContainer" containerID="b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b" Oct 31 13:45:14.768282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.768264 5074 scope.go:122] "RemoveContainer" containerID="b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b" Oct 31 13:45:14.768559 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:14.768537 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b\": container with ID starting with b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b not found: ID does not exist" containerID="b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b" Oct 31 13:45:14.768660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.768571 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b"} err="failed to get container status \"b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b\": rpc error: code = NotFound desc = could not find container \"b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b\": container with ID starting with b07b66177e74a72a3a01520105f78c076ab5d22e4a7c20c4864d7ef68185cf6b not found: ID does not exist" Oct 31 13:45:14.768660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.768594 5074 scope.go:122] "RemoveContainer" containerID="5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58" Oct 31 13:45:14.775789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.775763 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862"] Oct 31 13:45:14.779106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.779088 5074 scope.go:122] "RemoveContainer" containerID="5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58" Oct 31 13:45:14.779432 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:14.779410 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58\": container with ID starting with 5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58 not found: ID does not exist" containerID="5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58" Oct 31 13:45:14.779498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.779440 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58"} err="failed to get container status \"5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58\": rpc error: code = NotFound desc = could not find container \"5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58\": container with ID starting with 5e2df943d9594f79294a25ca835af3d105773820cde4196e90583c8ef17e6e58 not found: ID does not exist" Oct 31 13:45:14.779498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.779458 5074 scope.go:122] "RemoveContainer" containerID="8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6" Oct 31 13:45:14.780666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.780627 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-2-w3-47bcd9-kserve-6df5484588-dm862"] Oct 31 13:45:14.787348 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.787326 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg"] Oct 31 13:45:14.789722 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.789699 5074 scope.go:122] "RemoveContainer" containerID="8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6" Oct 31 13:45:14.790050 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:14.790028 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6\": container with ID starting with 8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6 not found: ID does not exist" containerID="8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6" Oct 31 13:45:14.790145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.790056 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6"} err="failed to get container status \"8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6\": rpc error: code = NotFound desc = could not find container \"8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6\": container with ID starting with 8b9062d9ea3bc22bc089ca3d1b5d428c2264d08565f19aef70dc91dcae6c50e6 not found: ID does not exist" Oct 31 13:45:14.792572 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.792550 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg"] Oct 31 13:45:14.804201 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.804180 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc"] Oct 31 13:45:14.810238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.810216 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc"] Oct 31 13:45:14.823015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.822988 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv"] Oct 31 13:45:14.826661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.826620 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-47bcd9/premium-w3-47bcd9-kserve-585b654656-7v5xv"] Oct 31 13:45:14.931466 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:14.931433 5074 scope.go:122] "RemoveContainer" containerID="c8e170eea1c8bf865ab036a1be4ed458e074e7279b294d0d2ecd781f2a701808" Oct 31 13:45:15.087305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.087272 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:45:15.258665 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.258549 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/simulator-w3-47bcd9-kserve-79f9d74765-5gtgc" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:45:15.258812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.258569 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-47bcd9/distinct-w3-47bcd9-kserve-65c66584c5-dbqwg" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:45:15.757648 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.757589 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" event={"ID":"37bda705-a0e5-4214-b53d-557561d0a67f","Type":"ContainerStarted","Data":"9da7c211cc499c676839964e24148e5a1453f8e7f5b4fdb4292591adb2ddf250"} Oct 31 13:45:15.940105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.940069 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" path="/var/lib/kubelet/pods/296a60e8-1efa-44da-bed6-842ccefdd93d/volumes" Oct 31 13:45:15.940516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.940501 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" path="/var/lib/kubelet/pods/66c03650-44c5-4c06-b8a3-940deaa36257/volumes" Oct 31 13:45:15.940942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.940926 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" path="/var/lib/kubelet/pods/90a85d43-3cca-48ff-b8a3-f8de2beff5b1/volumes" Oct 31 13:45:15.941577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:15.941562 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" path="/var/lib/kubelet/pods/d94beb7c-137f-4c8c-b08e-cf95d981f725/volumes" Oct 31 13:45:16.765854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:16.765815 5074 generic.go:356] "Generic (PLEG): container finished" podID="37bda705-a0e5-4214-b53d-557561d0a67f" containerID="9da7c211cc499c676839964e24148e5a1453f8e7f5b4fdb4292591adb2ddf250" exitCode=60 Oct 31 13:45:16.766354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:16.765892 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" event={"ID":"37bda705-a0e5-4214-b53d-557561d0a67f","Type":"ContainerDied","Data":"9da7c211cc499c676839964e24148e5a1453f8e7f5b4fdb4292591adb2ddf250"} Oct 31 13:45:16.766354 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:16.765956 5074 scope.go:122] "RemoveContainer" containerID="c8e170eea1c8bf865ab036a1be4ed458e074e7279b294d0d2ecd781f2a701808" Oct 31 13:45:16.766507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:16.766414 5074 scope.go:122] "RemoveContainer" containerID="9da7c211cc499c676839964e24148e5a1453f8e7f5b4fdb4292591adb2ddf250" Oct 31 13:45:16.766766 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:16.766742 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj_odh-ai-gateway-infra(37bda705-a0e5-4214-b53d-557561d0a67f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" Oct 31 13:45:16.805164 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:16.805136 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj"] Oct 31 13:45:17.918273 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:17.918246 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:17.931906 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:17.931852 5074 scope.go:122] "RemoveContainer" containerID="3d8a4d4f0f59a8b986fbdbcdd725040dadc1e20b44ae2a7e5bc864e1a894af17" Oct 31 13:45:17.932076 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:17.931948 5074 scope.go:122] "RemoveContainer" containerID="eea28a63d3ff2ed23fe51bbad149479db45bc32657950dfb790a9f8a65374520" Oct 31 13:45:17.932735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:17.932479 5074 scope.go:122] "RemoveContainer" containerID="3fe63b38ee6caf631b9024368a4a43d58598d402f1fac9e8624dd0591a98102f" Oct 31 13:45:18.044310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.044284 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bda705-a0e5-4214-b53d-557561d0a67f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bda705-a0e5-4214-b53d-557561d0a67f-maas-api-service-ca\") pod \"37bda705-a0e5-4214-b53d-557561d0a67f\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " Oct 31 13:45:18.044433 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.044396 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bda705-a0e5-4214-b53d-557561d0a67f-kube-api-access-7vqgv\" (UniqueName: \"kubernetes.io/projected/37bda705-a0e5-4214-b53d-557561d0a67f-kube-api-access-7vqgv\") pod \"37bda705-a0e5-4214-b53d-557561d0a67f\" (UID: \"37bda705-a0e5-4214-b53d-557561d0a67f\") " Oct 31 13:45:18.044671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.044641 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bda705-a0e5-4214-b53d-557561d0a67f-maas-api-service-ca" pod "37bda705-a0e5-4214-b53d-557561d0a67f" (UID: "37bda705-a0e5-4214-b53d-557561d0a67f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:18.044991 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.044969 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bda705-a0e5-4214-b53d-557561d0a67f-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:18.046455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.046428 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bda705-a0e5-4214-b53d-557561d0a67f-kube-api-access-7vqgv" pod "37bda705-a0e5-4214-b53d-557561d0a67f" (UID: "37bda705-a0e5-4214-b53d-557561d0a67f"). InnerVolumeSpecName "kube-api-access-7vqgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:18.146046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.146008 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqgv\" (UniqueName: \"kubernetes.io/projected/37bda705-a0e5-4214-b53d-557561d0a67f-kube-api-access-7vqgv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:18.784427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.784390 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" Oct 31 13:45:18.784603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.784415 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj" event={"ID":"37bda705-a0e5-4214-b53d-557561d0a67f","Type":"ContainerDied","Data":"ad11a8787477cdd3b5f0763b4842c529c82a4a1d25c3893f009ee1d8b410afe6"} Oct 31 13:45:18.784603 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.784521 5074 scope.go:122] "RemoveContainer" containerID="9da7c211cc499c676839964e24148e5a1453f8e7f5b4fdb4292591adb2ddf250" Oct 31 13:45:18.787973 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.787127 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerStarted","Data":"b867dc32073d4a584aa16d1c17ab4a7cf4eb14db8b7f678ebb4c85d2ed6c552a"} Oct 31 13:45:18.789535 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.789507 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerStarted","Data":"8bce9eab7ebba81794a302e8dec8349152a78be431b360ff004519ef6cb67c46"} Oct 31 13:45:18.791824 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.791802 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerStarted","Data":"361c7ec7cc970b54a482502fc6a8eddc18ca8c808a7e72d59b067c70ca842cf0"} Oct 31 13:45:18.824291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.824246 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj"] Oct 31 13:45:18.828687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:18.828651 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-47bcd9-29890905-2cvkj"] Oct 31 13:45:19.801131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.801095 5074 generic.go:356] "Generic (PLEG): container finished" podID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerID="361c7ec7cc970b54a482502fc6a8eddc18ca8c808a7e72d59b067c70ca842cf0" exitCode=60 Oct 31 13:45:19.801672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.801168 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerDied","Data":"361c7ec7cc970b54a482502fc6a8eddc18ca8c808a7e72d59b067c70ca842cf0"} Oct 31 13:45:19.801672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.801232 5074 scope.go:122] "RemoveContainer" containerID="3d8a4d4f0f59a8b986fbdbcdd725040dadc1e20b44ae2a7e5bc864e1a894af17" Oct 31 13:45:19.801672 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.801561 5074 scope.go:122] "RemoveContainer" containerID="361c7ec7cc970b54a482502fc6a8eddc18ca8c808a7e72d59b067c70ca842cf0" Oct 31 13:45:19.801899 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:19.801868 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh_odh-ai-gateway-infra(fa537dee-f5b5-4c47-932f-3c9ee78d9c8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" Oct 31 13:45:19.804244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.804218 5074 generic.go:356] "Generic (PLEG): container finished" podID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerID="b867dc32073d4a584aa16d1c17ab4a7cf4eb14db8b7f678ebb4c85d2ed6c552a" exitCode=60 Oct 31 13:45:19.804369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.804286 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerDied","Data":"b867dc32073d4a584aa16d1c17ab4a7cf4eb14db8b7f678ebb4c85d2ed6c552a"} Oct 31 13:45:19.804618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.804600 5074 scope.go:122] "RemoveContainer" containerID="b867dc32073d4a584aa16d1c17ab4a7cf4eb14db8b7f678ebb4c85d2ed6c552a" Oct 31 13:45:19.804888 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:19.804869 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2_odh-ai-gateway-infra(fc96ebae-a48c-4fff-8192-a013c6eaaead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" Oct 31 13:45:19.806451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.806426 5074 generic.go:356] "Generic (PLEG): container finished" podID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerID="8bce9eab7ebba81794a302e8dec8349152a78be431b360ff004519ef6cb67c46" exitCode=60 Oct 31 13:45:19.806568 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.806490 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerDied","Data":"8bce9eab7ebba81794a302e8dec8349152a78be431b360ff004519ef6cb67c46"} Oct 31 13:45:19.806825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.806810 5074 scope.go:122] "RemoveContainer" containerID="8bce9eab7ebba81794a302e8dec8349152a78be431b360ff004519ef6cb67c46" Oct 31 13:45:19.807044 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:19.807026 5074 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc_odh-ai-gateway-infra(d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" Oct 31 13:45:19.818247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.818103 5074 scope.go:122] "RemoveContainer" containerID="eea28a63d3ff2ed23fe51bbad149479db45bc32657950dfb790a9f8a65374520" Oct 31 13:45:19.822387 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.822366 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2"] Oct 31 13:45:19.832695 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.832677 5074 scope.go:122] "RemoveContainer" containerID="3fe63b38ee6caf631b9024368a4a43d58598d402f1fac9e8624dd0591a98102f" Oct 31 13:45:19.875675 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.869240 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc"] Oct 31 13:45:19.883072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.883038 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh"] Oct 31 13:45:19.938271 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:19.938237 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" path="/var/lib/kubelet/pods/37bda705-a0e5-4214-b53d-557561d0a67f/volumes" Oct 31 13:45:21.021069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.021038 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:21.064382 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.064353 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:21.067617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.067564 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:21.184531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.184370 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-maas-api-service-ca\") pod \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " Oct 31 13:45:21.184531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.184441 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc96ebae-a48c-4fff-8192-a013c6eaaead-kube-api-access-n4fk5\" (UniqueName: \"kubernetes.io/projected/fc96ebae-a48c-4fff-8192-a013c6eaaead-kube-api-access-n4fk5\") pod \"fc96ebae-a48c-4fff-8192-a013c6eaaead\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " Oct 31 13:45:21.184878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.184737 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-kube-api-access-kmwbx\" (UniqueName: \"kubernetes.io/projected/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-kube-api-access-kmwbx\") pod \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\" (UID: \"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e\") " Oct 31 13:45:21.184878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.184865 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-kube-api-access-9ml2g\" (UniqueName: \"kubernetes.io/projected/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-kube-api-access-9ml2g\") pod \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " Oct 31 13:45:21.185014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.184918 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc96ebae-a48c-4fff-8192-a013c6eaaead-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc96ebae-a48c-4fff-8192-a013c6eaaead-maas-api-service-ca\") pod \"fc96ebae-a48c-4fff-8192-a013c6eaaead\" (UID: \"fc96ebae-a48c-4fff-8192-a013c6eaaead\") " Oct 31 13:45:21.185014 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.184967 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-maas-api-service-ca\") pod \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\" (UID: \"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9\") " Oct 31 13:45:21.189445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.186258 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-maas-api-service-ca" pod "d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" (UID: "d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.189445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.186609 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc96ebae-a48c-4fff-8192-a013c6eaaead-maas-api-service-ca" pod "fc96ebae-a48c-4fff-8192-a013c6eaaead" (UID: "fc96ebae-a48c-4fff-8192-a013c6eaaead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.189445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.186872 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-maas-api-service-ca" pod "fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" (UID: "fa537dee-f5b5-4c47-932f-3c9ee78d9c8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.189445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.189113 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-kube-api-access-9ml2g" pod "d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" (UID: "d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9"). InnerVolumeSpecName "kube-api-access-9ml2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.208827 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.206850 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc96ebae-a48c-4fff-8192-a013c6eaaead-kube-api-access-n4fk5" pod "fc96ebae-a48c-4fff-8192-a013c6eaaead" (UID: "fc96ebae-a48c-4fff-8192-a013c6eaaead"). InnerVolumeSpecName "kube-api-access-n4fk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.209528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.209487 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-kube-api-access-kmwbx" pod "fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" (UID: "fa537dee-f5b5-4c47-932f-3c9ee78d9c8e"). InnerVolumeSpecName "kube-api-access-kmwbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.287854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.287807 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4fk5\" (UniqueName: \"kubernetes.io/projected/fc96ebae-a48c-4fff-8192-a013c6eaaead-kube-api-access-n4fk5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.287854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.287849 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwbx\" (UniqueName: \"kubernetes.io/projected/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-kube-api-access-kmwbx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.288120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.287867 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ml2g\" (UniqueName: \"kubernetes.io/projected/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-kube-api-access-9ml2g\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.288120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.287882 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc96ebae-a48c-4fff-8192-a013c6eaaead-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.288120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.287898 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.288120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.287913 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e-maas-api-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.826238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.826204 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" Oct 31 13:45:21.826429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.826202 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2" event={"ID":"fc96ebae-a48c-4fff-8192-a013c6eaaead","Type":"ContainerDied","Data":"840a9ce847e95a07fa9acf46c2d0652ff4322ce070cc2ba01932962b2ac1c4d1"} Oct 31 13:45:21.826429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.826380 5074 scope.go:122] "RemoveContainer" containerID="b867dc32073d4a584aa16d1c17ab4a7cf4eb14db8b7f678ebb4c85d2ed6c552a" Oct 31 13:45:21.828002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.827693 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" Oct 31 13:45:21.828002 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.827693 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc" event={"ID":"d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9","Type":"ContainerDied","Data":"b36f220a3fdeb0723b0e53656e7dd02ab23c6fd70913169307f0b227ddd1a144"} Oct 31 13:45:21.831152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.831122 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" event={"ID":"fa537dee-f5b5-4c47-932f-3c9ee78d9c8e","Type":"ContainerDied","Data":"ea5ee748d894a2c969318ce4782a84021650719fe9421f53bb7c234fee672b7b"} Oct 31 13:45:21.831339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.831132 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh" Oct 31 13:45:21.839786 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.839763 5074 scope.go:122] "RemoveContainer" containerID="8bce9eab7ebba81794a302e8dec8349152a78be431b360ff004519ef6cb67c46" Oct 31 13:45:21.850333 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.850309 5074 scope.go:122] "RemoveContainer" containerID="361c7ec7cc970b54a482502fc6a8eddc18ca8c808a7e72d59b067c70ca842cf0" Oct 31 13:45:21.856501 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.856235 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2"] Oct 31 13:45:21.861175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.861149 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0612-29890905-rfmx2"] Oct 31 13:45:21.879965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.879936 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc"] Oct 31 13:45:21.886835 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.886809 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1e72f6-29890905-9nrpc"] Oct 31 13:45:21.897088 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.897064 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh"] Oct 31 13:45:21.903123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.903096 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15a6a2-29890905-7flqh"] Oct 31 13:45:21.938158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.938126 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" path="/var/lib/kubelet/pods/d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9/volumes" Oct 31 13:45:21.938643 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.938614 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" path="/var/lib/kubelet/pods/fa537dee-f5b5-4c47-932f-3c9ee78d9c8e/volumes" Oct 31 13:45:21.939047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:21.939032 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" path="/var/lib/kubelet/pods/fc96ebae-a48c-4fff-8192-a013c6eaaead/volumes" Oct 31 13:45:25.983466 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:25.983431 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:45:31.080860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:31.080817 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck"] Oct 31 13:45:33.076467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:33.076430 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:45:38.029838 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:38.029799 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs"] Oct 31 13:45:38.809615 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:38.809573 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z"] Oct 31 13:45:38.809874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:38.809848 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" containerID="cri-o://3cd3da5904addaf8e3e784eba2d8aa0b4dbd99436a3d403339de23f1a94b83bd" gracePeriod=30 Oct 31 13:45:43.030467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:43.030406 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" podUID="703441e7-a964-49b8-8af3-8c951959bd22" containerName="payload-pre-processing" containerID="cri-o://ba936571746ed2d3be1ca4406bb87f54f5cc676e507401af1d65ed393cf451de" gracePeriod=25 Oct 31 13:45:44.013529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.013460 5074 generic.go:356] "Generic (PLEG): container finished" podID="c6ef088b-8828-4966-9457-fe4346c638b1" containerID="3cd3da5904addaf8e3e784eba2d8aa0b4dbd99436a3d403339de23f1a94b83bd" exitCode=0 Oct 31 13:45:44.013775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.013523 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" event={"ID":"c6ef088b-8828-4966-9457-fe4346c638b1","Type":"ContainerDied","Data":"3cd3da5904addaf8e3e784eba2d8aa0b4dbd99436a3d403339de23f1a94b83bd"} Oct 31 13:45:44.072162 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.072135 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 13:45:44.101594 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101520 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-kube-api-access-kl5qh\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-kube-api-access-kl5qh\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.101768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101610 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-istio-token\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-istio-token\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.101768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101662 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-credential-socket\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.101768 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101725 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-envoy\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.101942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101782 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-certs\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.101942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101825 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6ef088b-8828-4966-9457-fe4346c638b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ef088b-8828-4966-9457-fe4346c638b1-istio-podinfo\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.101942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101867 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-data\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.101942 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101930 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ef088b-8828-4966-9457-fe4346c638b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ef088b-8828-4966-9457-fe4346c638b1-istiod-ca-cert\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.102189 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.101956 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-socket\") pod \"c6ef088b-8828-4966-9457-fe4346c638b1\" (UID: \"c6ef088b-8828-4966-9457-fe4346c638b1\") " Oct 31 13:45:44.103202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.102680 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-socket" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.103202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.102917 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-data" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.103515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.103200 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ef088b-8828-4966-9457-fe4346c638b1-istiod-ca-cert" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:44.103515 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.103478 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-certs" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.103705 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.103681 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-credential-socket" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.106025 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.105999 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-kube-api-access-kl5qh" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "kube-api-access-kl5qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:44.106410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.106357 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-envoy" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:44.106497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.106466 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6ef088b-8828-4966-9457-fe4346c638b1-istio-podinfo" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:44.106497 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.106480 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-istio-token" pod "c6ef088b-8828-4966-9457-fe4346c638b1" (UID: "c6ef088b-8828-4966-9457-fe4346c638b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:44.203822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203780 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.203822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203824 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.204034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203840 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.204034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203858 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.204034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203874 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ef088b-8828-4966-9457-fe4346c638b1-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.204034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203888 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.204034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203903 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ef088b-8828-4966-9457-fe4346c638b1-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.204034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203918 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ef088b-8828-4966-9457-fe4346c638b1-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:44.204034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:44.203932 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl5qh\" (UniqueName: \"kubernetes.io/projected/c6ef088b-8828-4966-9457-fe4346c638b1-kube-api-access-kl5qh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:45.022196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:45.022163 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" Oct 31 13:45:45.022376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:45.022170 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z" event={"ID":"c6ef088b-8828-4966-9457-fe4346c638b1","Type":"ContainerDied","Data":"21d29c7af0c064f9980fba8f8423bac48815d513c511f1fad75fcab06e0049ca"} Oct 31 13:45:45.022376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:45.022326 5074 scope.go:122] "RemoveContainer" containerID="3cd3da5904addaf8e3e784eba2d8aa0b4dbd99436a3d403339de23f1a94b83bd" Oct 31 13:45:45.049794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:45.049769 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z"] Oct 31 13:45:45.056460 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:45.056433 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-47bcd9-openshift-default-b88686545-sdl9z"] Oct 31 13:45:45.680202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:45.680161 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" podUID="703441e7-a964-49b8-8af3-8c951959bd22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:45.938080 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:45.937985 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" path="/var/lib/kubelet/pods/c6ef088b-8828-4966-9457-fe4346c638b1/volumes" Oct 31 13:45:46.103248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.103195 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" podUID="2dd55587-a53c-4c0e-8642-15260308824a" containerName="main" containerID="cri-o://eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b" gracePeriod=45 Oct 31 13:45:46.373425 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.373401 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:45:46.429798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.429764 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-tmp-dir\") pod \"2dd55587-a53c-4c0e-8642-15260308824a\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " Oct 31 13:45:46.429975 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.429924 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd55587-a53c-4c0e-8642-15260308824a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd55587-a53c-4c0e-8642-15260308824a-tls-certs\") pod \"2dd55587-a53c-4c0e-8642-15260308824a\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " Oct 31 13:45:46.429975 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.429955 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-dshm\") pod \"2dd55587-a53c-4c0e-8642-15260308824a\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " Oct 31 13:45:46.430133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.429996 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd55587-a53c-4c0e-8642-15260308824a-kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/2dd55587-a53c-4c0e-8642-15260308824a-kube-api-access-nn4zn\") pod \"2dd55587-a53c-4c0e-8642-15260308824a\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " Oct 31 13:45:46.430133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.430060 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-home\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-home\") pod \"2dd55587-a53c-4c0e-8642-15260308824a\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " Oct 31 13:45:46.430133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.430105 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-tmp-dir" pod "2dd55587-a53c-4c0e-8642-15260308824a" (UID: "2dd55587-a53c-4c0e-8642-15260308824a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:46.430296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.430132 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-model-cache\") pod \"2dd55587-a53c-4c0e-8642-15260308824a\" (UID: \"2dd55587-a53c-4c0e-8642-15260308824a\") " Oct 31 13:45:46.430361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.430346 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-home" pod "2dd55587-a53c-4c0e-8642-15260308824a" (UID: "2dd55587-a53c-4c0e-8642-15260308824a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:46.430546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.430524 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-model-cache" pod "2dd55587-a53c-4c0e-8642-15260308824a" (UID: "2dd55587-a53c-4c0e-8642-15260308824a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:46.430616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.430533 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:46.430616 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.430560 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:46.432339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.432313 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd55587-a53c-4c0e-8642-15260308824a-tls-certs" pod "2dd55587-a53c-4c0e-8642-15260308824a" (UID: "2dd55587-a53c-4c0e-8642-15260308824a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:46.432445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.432359 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd55587-a53c-4c0e-8642-15260308824a-kube-api-access-nn4zn" pod "2dd55587-a53c-4c0e-8642-15260308824a" (UID: "2dd55587-a53c-4c0e-8642-15260308824a"). InnerVolumeSpecName "kube-api-access-nn4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:46.432583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.432562 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-dshm" pod "2dd55587-a53c-4c0e-8642-15260308824a" (UID: "2dd55587-a53c-4c0e-8642-15260308824a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:46.531624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.531548 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd55587-a53c-4c0e-8642-15260308824a-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:46.531624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.531586 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:46.531624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.531602 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/2dd55587-a53c-4c0e-8642-15260308824a-kube-api-access-nn4zn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:46.531624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:46.531618 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd55587-a53c-4c0e-8642-15260308824a-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:47.040676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.040613 5074 generic.go:356] "Generic (PLEG): container finished" podID="2dd55587-a53c-4c0e-8642-15260308824a" containerID="eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b" exitCode=0 Oct 31 13:45:47.041151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.040676 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" event={"ID":"2dd55587-a53c-4c0e-8642-15260308824a","Type":"ContainerDied","Data":"eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b"} Oct 31 13:45:47.041151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.040688 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" Oct 31 13:45:47.041151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.040722 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck" event={"ID":"2dd55587-a53c-4c0e-8642-15260308824a","Type":"ContainerDied","Data":"749be560d495c05830bab914677c8f25b063e883e31a1639ed5542415ea756b3"} Oct 31 13:45:47.041151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.040742 5074 scope.go:122] "RemoveContainer" containerID="eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b" Oct 31 13:45:47.060207 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.060180 5074 scope.go:122] "RemoveContainer" containerID="eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b" Oct 31 13:45:47.060525 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:47.060504 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b\": container with ID starting with eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b not found: ID does not exist" containerID="eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b" Oct 31 13:45:47.060673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.060533 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b"} err="failed to get container status \"eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b\": rpc error: code = NotFound desc = could not find container \"eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b\": container with ID starting with eabfe89a703a4487f2407251c57bcf61cbabd45ed12233cc89c8cd49a5da686b not found: ID does not exist" Oct 31 13:45:47.077564 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.077537 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck"] Oct 31 13:45:47.081988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.081959 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1e72f6/sub-test-model-1e72f6-kserve-5785ccf867-nvnck"] Oct 31 13:45:47.938301 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:47.938255 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd55587-a53c-4c0e-8642-15260308824a" path="/var/lib/kubelet/pods/2dd55587-a53c-4c0e-8642-15260308824a/volumes" Oct 31 13:45:48.296788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:48.296746 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:45:48.482431 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:48.482397 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2"] Oct 31 13:45:48.482732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:48.482667 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" containerID="cri-o://2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950" gracePeriod=30 Oct 31 13:45:53.098529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.098491 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp"] Oct 31 13:45:53.105147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.105118 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4d5878-r77vp"] Oct 31 13:45:53.744882 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.744857 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:45:53.814939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.814836 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-podinfo\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.814939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.814904 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-socket\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.814951 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-credential-socket\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.814986 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-envoy\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815035 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-token\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815073 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-data\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815137 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-certs\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815258 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a2814f-e408-4a17-9e98-39c7227ee416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2814f-e408-4a17-9e98-39c7227ee416-istiod-ca-cert\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815273 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-socket" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:53.815505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815303 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-kube-api-access-xwjbc\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-kube-api-access-xwjbc\") pod \"a7a2814f-e408-4a17-9e98-39c7227ee416\" (UID: \"a7a2814f-e408-4a17-9e98-39c7227ee416\") " Oct 31 13:45:53.815505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815494 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-credential-socket" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:53.815775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815677 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-certs" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:53.815775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815720 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.815775 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.815742 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.816308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.816074 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a2814f-e408-4a17-9e98-39c7227ee416-istiod-ca-cert" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:53.816308 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.816260 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-data" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:53.818752 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.818723 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-envoy" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:53.818854 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.818812 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-kube-api-access-xwjbc" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "kube-api-access-xwjbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:53.819152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.819126 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-token" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:53.819343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.819324 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-podinfo" pod "a7a2814f-e408-4a17-9e98-39c7227ee416" (UID: "a7a2814f-e408-4a17-9e98-39c7227ee416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:53.917209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.917170 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2814f-e408-4a17-9e98-39c7227ee416-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.917209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.917211 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjbc\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-kube-api-access-xwjbc\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.917397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.917228 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.917397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.917243 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.917397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.917258 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.917397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.917272 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.917397 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.917285 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a2814f-e408-4a17-9e98-39c7227ee416-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.938155 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:53.938120 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f046502-f686-4bc5-97cb-8b568b9c6d9d" path="/var/lib/kubelet/pods/7f046502-f686-4bc5-97cb-8b568b9c6d9d/volumes" Oct 31 13:45:54.092209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.092112 5074 generic.go:356] "Generic (PLEG): container finished" podID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerID="2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950" exitCode=0 Oct 31 13:45:54.092209 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.092166 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" Oct 31 13:45:54.092455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.092169 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" event={"ID":"a7a2814f-e408-4a17-9e98-39c7227ee416","Type":"ContainerDied","Data":"2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950"} Oct 31 13:45:54.092455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.092296 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2" event={"ID":"a7a2814f-e408-4a17-9e98-39c7227ee416","Type":"ContainerDied","Data":"ddd57720adf98b12bceab3962d81305a78e210205db0bcaf2ab62a38363292e7"} Oct 31 13:45:54.092455 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.092320 5074 scope.go:122] "RemoveContainer" containerID="2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950" Oct 31 13:45:54.104998 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.104975 5074 scope.go:122] "RemoveContainer" containerID="2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950" Oct 31 13:45:54.105325 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:54.105258 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950\": container with ID starting with 2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950 not found: ID does not exist" containerID="2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950" Oct 31 13:45:54.105325 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.105282 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950"} err="failed to get container status \"2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950\": rpc error: code = NotFound desc = could not find container \"2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950\": container with ID starting with 2584d1979d33c7e850a246255dcc087300d6f5f82f913d7e9c3038cf7090b950 not found: ID does not exist" Oct 31 13:45:54.134652 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.116857 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2"] Oct 31 13:45:54.143771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:54.143738 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-842ac9-openshift-default-56c799f9cc-dsrv2"] Oct 31 13:45:55.162871 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:55.162840 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9"] Oct 31 13:45:55.185281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:55.185246 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2"] Oct 31 13:45:55.680430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:55.680385 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" podUID="703441e7-a964-49b8-8af3-8c951959bd22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:45:55.937872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:55.937793 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" path="/var/lib/kubelet/pods/a7a2814f-e408-4a17-9e98-39c7227ee416/volumes" Oct 31 13:45:55.982100 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:45:55.982064 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:45:59.404462 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:45:59.404422 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:46:00.163864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:00.163815 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" podUID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" containerName="payload-processing" containerID="cri-o://ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a" gracePeriod=25 Oct 31 13:46:00.185803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:00.185751 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" podUID="38f511f1-b31c-44aa-a6de-04741bfe6f08" containerName="payload-pre-processing" containerID="cri-o://e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82" gracePeriod=25 Oct 31 13:46:03.150371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.150334 5074 generic.go:356] "Generic (PLEG): container finished" podID="703441e7-a964-49b8-8af3-8c951959bd22" containerID="ba936571746ed2d3be1ca4406bb87f54f5cc676e507401af1d65ed393cf451de" exitCode=0 Oct 31 13:46:03.150628 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.150455 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" event={"ID":"703441e7-a964-49b8-8af3-8c951959bd22","Type":"ContainerDied","Data":"ba936571746ed2d3be1ca4406bb87f54f5cc676e507401af1d65ed393cf451de"} Oct 31 13:46:03.293340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.293273 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:46:03.426938 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.426897 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703441e7-a964-49b8-8af3-8c951959bd22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703441e7-a964-49b8-8af3-8c951959bd22-plugins-config-volume\") pod \"703441e7-a964-49b8-8af3-8c951959bd22\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " Oct 31 13:46:03.427158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.426993 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703441e7-a964-49b8-8af3-8c951959bd22-kube-api-access-jm562\" (UniqueName: \"kubernetes.io/projected/703441e7-a964-49b8-8af3-8c951959bd22-kube-api-access-jm562\") pod \"703441e7-a964-49b8-8af3-8c951959bd22\" (UID: \"703441e7-a964-49b8-8af3-8c951959bd22\") " Oct 31 13:46:03.427332 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.427304 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703441e7-a964-49b8-8af3-8c951959bd22-plugins-config-volume" pod "703441e7-a964-49b8-8af3-8c951959bd22" (UID: "703441e7-a964-49b8-8af3-8c951959bd22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:03.429087 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.429068 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703441e7-a964-49b8-8af3-8c951959bd22-kube-api-access-jm562" pod "703441e7-a964-49b8-8af3-8c951959bd22" (UID: "703441e7-a964-49b8-8af3-8c951959bd22"). InnerVolumeSpecName "kube-api-access-jm562". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:03.528293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.528248 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703441e7-a964-49b8-8af3-8c951959bd22-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:03.528293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:03.528288 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm562\" (UniqueName: \"kubernetes.io/projected/703441e7-a964-49b8-8af3-8c951959bd22-kube-api-access-jm562\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:04.158142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:04.158099 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" Oct 31 13:46:04.158660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:04.158109 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs" event={"ID":"703441e7-a964-49b8-8af3-8c951959bd22","Type":"ContainerDied","Data":"3cbefd558e1f2b93eb8f6c158187ef2598563732364c1a15d93d729a341e1cd8"} Oct 31 13:46:04.158660 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:04.158257 5074 scope.go:122] "RemoveContainer" containerID="ba936571746ed2d3be1ca4406bb87f54f5cc676e507401af1d65ed393cf451de" Oct 31 13:46:04.180958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:04.180929 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs"] Oct 31 13:46:04.193565 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:04.192345 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-842ac9-66f8c75cff-l9cxs"] Oct 31 13:46:05.034043 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:05.033997 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb"] Oct 31 13:46:05.039011 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:05.038975 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d345050-m78cb"] Oct 31 13:46:05.937485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:05.937449 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703441e7-a964-49b8-8af3-8c951959bd22" path="/var/lib/kubelet/pods/703441e7-a964-49b8-8af3-8c951959bd22/volumes" Oct 31 13:46:05.937946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:05.937930 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f" path="/var/lib/kubelet/pods/7eeffeb1-bb62-41de-b0c6-805aa3ed1f4f/volumes" Oct 31 13:46:07.342607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:07.342556 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" podUID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:07.342607 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:07.342588 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" podUID="38f511f1-b31c-44aa-a6de-04741bfe6f08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:17.342144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:17.342095 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" podUID="38f511f1-b31c-44aa-a6de-04741bfe6f08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:17.342576 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:17.342095 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" podUID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:17.415516 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:17.415469 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:46:20.458468 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.458435 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:46:20.514281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.514248 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:46:20.520498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.520462 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5529ab24-7af7-4584-94cf-c6a6f485cd3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529ab24-7af7-4584-94cf-c6a6f485cd3f-plugins-config-volume\") pod \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " Oct 31 13:46:20.520673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.520580 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5529ab24-7af7-4584-94cf-c6a6f485cd3f-kube-api-access-cgb62\" (UniqueName: \"kubernetes.io/projected/5529ab24-7af7-4584-94cf-c6a6f485cd3f-kube-api-access-cgb62\") pod \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\" (UID: \"5529ab24-7af7-4584-94cf-c6a6f485cd3f\") " Oct 31 13:46:20.521101 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.521076 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5529ab24-7af7-4584-94cf-c6a6f485cd3f-plugins-config-volume" pod "5529ab24-7af7-4584-94cf-c6a6f485cd3f" (UID: "5529ab24-7af7-4584-94cf-c6a6f485cd3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:20.523113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.523083 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5529ab24-7af7-4584-94cf-c6a6f485cd3f-kube-api-access-cgb62" pod "5529ab24-7af7-4584-94cf-c6a6f485cd3f" (UID: "5529ab24-7af7-4584-94cf-c6a6f485cd3f"). InnerVolumeSpecName "kube-api-access-cgb62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:20.621558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.621525 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f511f1-b31c-44aa-a6de-04741bfe6f08-kube-api-access-nw2wh\" (UniqueName: \"kubernetes.io/projected/38f511f1-b31c-44aa-a6de-04741bfe6f08-kube-api-access-nw2wh\") pod \"38f511f1-b31c-44aa-a6de-04741bfe6f08\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " Oct 31 13:46:20.621558 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.621565 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f511f1-b31c-44aa-a6de-04741bfe6f08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f511f1-b31c-44aa-a6de-04741bfe6f08-plugins-config-volume\") pod \"38f511f1-b31c-44aa-a6de-04741bfe6f08\" (UID: \"38f511f1-b31c-44aa-a6de-04741bfe6f08\") " Oct 31 13:46:20.622021 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.621988 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgb62\" (UniqueName: \"kubernetes.io/projected/5529ab24-7af7-4584-94cf-c6a6f485cd3f-kube-api-access-cgb62\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:20.622104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.622020 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529ab24-7af7-4584-94cf-c6a6f485cd3f-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:20.622104 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.622045 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f511f1-b31c-44aa-a6de-04741bfe6f08-plugins-config-volume" pod "38f511f1-b31c-44aa-a6de-04741bfe6f08" (UID: "38f511f1-b31c-44aa-a6de-04741bfe6f08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:20.623743 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.623719 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f511f1-b31c-44aa-a6de-04741bfe6f08-kube-api-access-nw2wh" pod "38f511f1-b31c-44aa-a6de-04741bfe6f08" (UID: "38f511f1-b31c-44aa-a6de-04741bfe6f08"). InnerVolumeSpecName "kube-api-access-nw2wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:20.723126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.723091 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2wh\" (UniqueName: \"kubernetes.io/projected/38f511f1-b31c-44aa-a6de-04741bfe6f08-kube-api-access-nw2wh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:20.723126 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:20.723122 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f511f1-b31c-44aa-a6de-04741bfe6f08-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:21.264946 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.264904 5074 generic.go:356] "Generic (PLEG): container finished" podID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" containerID="ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a" exitCode=0 Oct 31 13:46:21.265152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.264956 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" Oct 31 13:46:21.265152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.264996 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" event={"ID":"5529ab24-7af7-4584-94cf-c6a6f485cd3f","Type":"ContainerDied","Data":"ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a"} Oct 31 13:46:21.265152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.265061 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9" event={"ID":"5529ab24-7af7-4584-94cf-c6a6f485cd3f","Type":"ContainerDied","Data":"eb2505aadde424c13a9b92855a8f099be6ae6b741b6b48b769c48c46bae34931"} Oct 31 13:46:21.265152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.265078 5074 scope.go:122] "RemoveContainer" containerID="ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a" Oct 31 13:46:21.267212 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.267177 5074 generic.go:356] "Generic (PLEG): container finished" podID="38f511f1-b31c-44aa-a6de-04741bfe6f08" containerID="e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82" exitCode=0 Oct 31 13:46:21.267339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.267210 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" event={"ID":"38f511f1-b31c-44aa-a6de-04741bfe6f08","Type":"ContainerDied","Data":"e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82"} Oct 31 13:46:21.267339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.267243 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" Oct 31 13:46:21.267339 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.267262 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2" event={"ID":"38f511f1-b31c-44aa-a6de-04741bfe6f08","Type":"ContainerDied","Data":"6b692b4b505b0c51503fea65b7b9f3c84dfedf0024b3827eaf3ebe9eb51e7b75"} Oct 31 13:46:21.276262 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.276225 5074 scope.go:122] "RemoveContainer" containerID="ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a" Oct 31 13:46:21.276916 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:46:21.276778 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a\": container with ID starting with ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a not found: ID does not exist" containerID="ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a" Oct 31 13:46:21.276916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.276820 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a"} err="failed to get container status \"ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a\": rpc error: code = NotFound desc = could not find container \"ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a\": container with ID starting with ad7632370180c3cb0cba37727a8ede2f679b634e8f5cbfddad3f8e6730b7af4a not found: ID does not exist" Oct 31 13:46:21.276916 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.276845 5074 scope.go:122] "RemoveContainer" containerID="e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82" Oct 31 13:46:21.286784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.286763 5074 scope.go:122] "RemoveContainer" containerID="e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82" Oct 31 13:46:21.287066 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:46:21.287041 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82\": container with ID starting with e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82 not found: ID does not exist" containerID="e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82" Oct 31 13:46:21.287131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.287076 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82"} err="failed to get container status \"e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82\": rpc error: code = NotFound desc = could not find container \"e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82\": container with ID starting with e678188ec32a0c1cdb7bf87de4a9570af252c678bf8de3a865c6e4ac4cb23e82 not found: ID does not exist" Oct 31 13:46:21.297894 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.297864 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9"] Oct 31 13:46:21.304187 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.304164 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15a6a2-6b555677bf-zmlv9"] Oct 31 13:46:21.318822 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.318789 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2"] Oct 31 13:46:21.330156 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.330130 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15a6a2-5fd9b56b67-j72l2"] Oct 31 13:46:21.938138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.938094 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f511f1-b31c-44aa-a6de-04741bfe6f08" path="/var/lib/kubelet/pods/38f511f1-b31c-44aa-a6de-04741bfe6f08/volumes" Oct 31 13:46:21.938699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:21.938677 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" path="/var/lib/kubelet/pods/5529ab24-7af7-4584-94cf-c6a6f485cd3f/volumes" Oct 31 13:46:24.067540 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:24.067504 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq"] Oct 31 13:46:24.068009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:24.067773 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" containerID="cri-o://7970aae06c487e6bbeea814effc79e7e40c5683ba353c514a1248932ffa67ba8" gracePeriod=30 Oct 31 13:46:25.983330 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:46:25.983292 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:46:29.322067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.322029 5074 generic.go:356] "Generic (PLEG): container finished" podID="83c12e2a-9e24-4adf-be3c-279947b46915" containerID="7970aae06c487e6bbeea814effc79e7e40c5683ba353c514a1248932ffa67ba8" exitCode=0 Oct 31 13:46:29.322422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.322096 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" event={"ID":"83c12e2a-9e24-4adf-be3c-279947b46915","Type":"ContainerDied","Data":"7970aae06c487e6bbeea814effc79e7e40c5683ba353c514a1248932ffa67ba8"} Oct 31 13:46:29.322422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.322153 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" event={"ID":"83c12e2a-9e24-4adf-be3c-279947b46915","Type":"ContainerDied","Data":"a7be44a1653e70200c1c22e15f247c8ab01b282b89c4c7e57d82ab594dee10ef"} Oct 31 13:46:29.322422 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.322170 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7be44a1653e70200c1c22e15f247c8ab01b282b89c4c7e57d82ab594dee10ef" Oct 31 13:46:29.331676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.331650 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:46:29.417751 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417647 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-socket\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.417751 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417741 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83c12e2a-9e24-4adf-be3c-279947b46915-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c12e2a-9e24-4adf-be3c-279947b46915-istio-podinfo\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417772 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-envoy\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417799 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-certs\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417836 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-istio-token\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-istio-token\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417912 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-kube-api-access-2zrss\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-kube-api-access-2zrss\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417950 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-credential-socket\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.417949 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-socket" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.418036 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418030 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c12e2a-9e24-4adf-be3c-279947b46915-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c12e2a-9e24-4adf-be3c-279947b46915-istiod-ca-cert\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418084 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-data\") pod \"83c12e2a-9e24-4adf-be3c-279947b46915\" (UID: \"83c12e2a-9e24-4adf-be3c-279947b46915\") " Oct 31 13:46:29.418448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418188 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-certs" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.418448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418430 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-data" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.418622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418469 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c12e2a-9e24-4adf-be3c-279947b46915-istiod-ca-cert" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:29.418622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418549 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.418622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418570 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.418622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418611 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.418853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.418739 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-credential-socket" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.420072 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.420043 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-envoy" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:29.420223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.420175 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c12e2a-9e24-4adf-be3c-279947b46915-istio-podinfo" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:29.420528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.420503 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-istio-token" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:29.420622 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.420522 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-kube-api-access-2zrss" pod "83c12e2a-9e24-4adf-be3c-279947b46915" (UID: "83c12e2a-9e24-4adf-be3c-279947b46915"). InnerVolumeSpecName "kube-api-access-2zrss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:29.520183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.520116 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.520183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.520150 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zrss\" (UniqueName: \"kubernetes.io/projected/83c12e2a-9e24-4adf-be3c-279947b46915-kube-api-access-2zrss\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.520183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.520160 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.520183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.520170 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c12e2a-9e24-4adf-be3c-279947b46915-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.520183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.520180 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c12e2a-9e24-4adf-be3c-279947b46915-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:29.520183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:29.520191 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c12e2a-9e24-4adf-be3c-279947b46915-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:30.328446 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:30.328406 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq" Oct 31 13:46:30.350617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:30.350583 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq"] Oct 31 13:46:30.356048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:30.356022 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0612-openshift-default-5cb97774f5-tf2tq"] Oct 31 13:46:31.937173 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:31.937132 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" path="/var/lib/kubelet/pods/83c12e2a-9e24-4adf-be3c-279947b46915/volumes" Oct 31 13:46:34.800261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:34.800219 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:46:40.199681 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:40.199620 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq"] Oct 31 13:46:40.200591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:40.200551 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" containerID="cri-o://b4824234367a5a2f72ef7ff84df4abf9d76a0c41d0c0ae7542c489619067c9cc" gracePeriod=30 Oct 31 13:46:45.169100 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.169062 5074 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:46:45.169496 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.169140 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:46:45.446321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.446286 5074 generic.go:356] "Generic (PLEG): container finished" podID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerID="b4824234367a5a2f72ef7ff84df4abf9d76a0c41d0c0ae7542c489619067c9cc" exitCode=0 Oct 31 13:46:45.446463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.446334 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" event={"ID":"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a","Type":"ContainerDied","Data":"b4824234367a5a2f72ef7ff84df4abf9d76a0c41d0c0ae7542c489619067c9cc"} Oct 31 13:46:45.446463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.446384 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" event={"ID":"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a","Type":"ContainerDied","Data":"12d954c73085790eae8cdd6b41cf3887a110a423870937cd5e6cc9360e6ca27b"} Oct 31 13:46:45.446463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.446407 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d954c73085790eae8cdd6b41cf3887a110a423870937cd5e6cc9360e6ca27b" Oct 31 13:46:45.455546 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.455520 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:46:45.586430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586388 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istiod-ca-cert\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586444 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-token\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586476 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-socket\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586582 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-data\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586695 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-certs\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586729 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-credential-socket\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586775 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-kube-api-access-st2bn\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-kube-api-access-st2bn\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586833 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-envoy\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.586872 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586854 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-socket" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.587158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586937 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-podinfo\") pod \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\" (UID: \"2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a\") " Oct 31 13:46:45.587158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586999 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-data" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.587158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.587014 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-certs" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.587158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.586911 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istiod-ca-cert" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:45.587158 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.587042 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-credential-socket" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.587398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.587321 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.587398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.587342 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.587398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.587356 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.587398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.587369 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.587398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.587382 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.589120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.589092 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-envoy" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:45.589230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.589126 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-podinfo" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:45.589230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.589165 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-kube-api-access-st2bn" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "kube-api-access-st2bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:45.589230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.589197 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-token" pod "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" (UID: "2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:45.680049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.679993 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n"] Oct 31 13:46:45.687979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.687914 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.687979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.687940 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st2bn\" (UniqueName: \"kubernetes.io/projected/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-kube-api-access-st2bn\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.687979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.687952 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:45.687979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:45.687961 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:46.452432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:46.452394 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq" Oct 31 13:46:46.476398 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:46.476346 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq"] Oct 31 13:46:46.483357 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:46.483321 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1e72f6-openshift-default-655d694775-jbcmq"] Oct 31 13:46:47.232427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:47.232398 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:46:47.260198 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:47.260168 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:46:47.682661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:47.682546 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:46:47.939139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:47.939056 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" path="/var/lib/kubelet/pods/2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a/volumes" Oct 31 13:46:53.555324 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:53.555282 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c"] Oct 31 13:46:53.555852 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:53.555531 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" containerID="cri-o://a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e" gracePeriod=30 Oct 31 13:46:55.036318 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:55.036263 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw"] Oct 31 13:46:55.039283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:55.039250 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e26e013-z9bmw"] Oct 31 13:46:55.938004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:55.937959 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62dc5df-b667-4645-b4ee-a9f37dc8c901" path="/var/lib/kubelet/pods/e62dc5df-b667-4645-b4ee-a9f37dc8c901/volumes" Oct 31 13:46:55.983309 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:46:55.983270 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:46:56.681027 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:56.680990 5074 scope.go:122] "RemoveContainer" containerID="e1438167de0b1a5f1dad71d2ee31c4a5fc51a6377b12be63a60d812d958500db" Oct 31 13:46:56.691136 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:56.691108 5074 scope.go:122] "RemoveContainer" containerID="b4824234367a5a2f72ef7ff84df4abf9d76a0c41d0c0ae7542c489619067c9cc" Oct 31 13:46:58.830170 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.830093 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:46:58.906860 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.906809 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-data\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.906916 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-envoy\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.906988 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-token\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-token\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907029 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istiod-ca-cert\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907079 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-kube-api-access-gpnbv\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-kube-api-access-gpnbv\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907112 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-credential-socket\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907163 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-socket\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907197 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-podinfo\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907241 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-certs\") pod \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\" (UID: \"845e6ed7-6c57-445f-acc3-8bd3f71bb72f\") " Oct 31 13:46:58.907902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907188 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-data" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:58.907902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907392 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-credential-socket" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:58.907902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907832 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-socket" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:58.907902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907785 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-certs" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:58.907902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.907887 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istiod-ca-cert" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:58.909593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.909555 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-envoy" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:58.909900 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.909863 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-podinfo" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:58.910039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.910001 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-token" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:58.910371 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:58.910342 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-kube-api-access-gpnbv" pod "845e6ed7-6c57-445f-acc3-8bd3f71bb72f" (UID: "845e6ed7-6c57-445f-acc3-8bd3f71bb72f"). InnerVolumeSpecName "kube-api-access-gpnbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:59.008676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008606 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpnbv\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-kube-api-access-gpnbv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008676 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008675 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008688 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008698 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008708 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008716 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008725 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008733 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.008914 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.008775 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845e6ed7-6c57-445f-acc3-8bd3f71bb72f-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:59.539102 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.539057 5074 generic.go:356] "Generic (PLEG): container finished" podID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerID="a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e" exitCode=0 Oct 31 13:46:59.539310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.539136 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" Oct 31 13:46:59.539310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.539161 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" event={"ID":"845e6ed7-6c57-445f-acc3-8bd3f71bb72f","Type":"ContainerDied","Data":"a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e"} Oct 31 13:46:59.539310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.539246 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c" event={"ID":"845e6ed7-6c57-445f-acc3-8bd3f71bb72f","Type":"ContainerDied","Data":"95050176605ed35bf1b78eebdfdc3624e648e280382b1b6d1b0b3b676c37fd3a"} Oct 31 13:46:59.539310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.539292 5074 scope.go:122] "RemoveContainer" containerID="a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e" Oct 31 13:46:59.550054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.550032 5074 scope.go:122] "RemoveContainer" containerID="a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e" Oct 31 13:46:59.550462 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:46:59.550373 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e\": container with ID starting with a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e not found: ID does not exist" containerID="a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e" Oct 31 13:46:59.550462 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.550411 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e"} err="failed to get container status \"a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e\": rpc error: code = NotFound desc = could not find container \"a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e\": container with ID starting with a70b1c56ba49b4b95820a43a03b5bc03ba0e61dbe501f9e4c989027bfbb4f10e not found: ID does not exist" Oct 31 13:46:59.572060 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.572023 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c"] Oct 31 13:46:59.603917 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.603871 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-2r62c"] Oct 31 13:46:59.937145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:46:59.937063 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" path="/var/lib/kubelet/pods/845e6ed7-6c57-445f-acc3-8bd3f71bb72f/volumes" Oct 31 13:47:00.701661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:00.701587 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" containerID="cri-o://1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2" gracePeriod=45 Oct 31 13:47:00.959623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:00.959552 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:47:01.027503 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.027465 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-model-cache\") pod \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " Oct 31 13:47:01.027735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.027543 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-home\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-home\") pod \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " Oct 31 13:47:01.027735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.027587 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-dshm\") pod \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " Oct 31 13:47:01.027735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.027660 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febb9a1e-921f-4e7d-9af7-a31a8113fbba-kube-api-access-6z4vd\" (UniqueName: \"kubernetes.io/projected/febb9a1e-921f-4e7d-9af7-a31a8113fbba-kube-api-access-6z4vd\") pod \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " Oct 31 13:47:01.027735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.027719 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tmp-dir\") pod \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " Oct 31 13:47:01.028005 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.027764 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tls-certs\" (UniqueName: \"kubernetes.io/secret/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tls-certs\") pod \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\" (UID: \"febb9a1e-921f-4e7d-9af7-a31a8113fbba\") " Oct 31 13:47:01.028667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.028623 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-model-cache" pod "febb9a1e-921f-4e7d-9af7-a31a8113fbba" (UID: "febb9a1e-921f-4e7d-9af7-a31a8113fbba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.028812 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.028794 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-home" pod "febb9a1e-921f-4e7d-9af7-a31a8113fbba" (UID: "febb9a1e-921f-4e7d-9af7-a31a8113fbba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.029291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.029246 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tmp-dir" pod "febb9a1e-921f-4e7d-9af7-a31a8113fbba" (UID: "febb9a1e-921f-4e7d-9af7-a31a8113fbba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.031792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.031755 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febb9a1e-921f-4e7d-9af7-a31a8113fbba-kube-api-access-6z4vd" pod "febb9a1e-921f-4e7d-9af7-a31a8113fbba" (UID: "febb9a1e-921f-4e7d-9af7-a31a8113fbba"). InnerVolumeSpecName "kube-api-access-6z4vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:01.036058 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.036033 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-dshm" pod "febb9a1e-921f-4e7d-9af7-a31a8113fbba" (UID: "febb9a1e-921f-4e7d-9af7-a31a8113fbba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:01.036254 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.036226 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tls-certs" pod "febb9a1e-921f-4e7d-9af7-a31a8113fbba" (UID: "febb9a1e-921f-4e7d-9af7-a31a8113fbba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:01.129388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.129350 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.129388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.129383 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.129388 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.129394 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z4vd\" (UniqueName: \"kubernetes.io/projected/febb9a1e-921f-4e7d-9af7-a31a8113fbba-kube-api-access-6z4vd\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.129668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.129405 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.129668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.129415 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febb9a1e-921f-4e7d-9af7-a31a8113fbba-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.129668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.129425 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febb9a1e-921f-4e7d-9af7-a31a8113fbba-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:01.553606 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.553565 5074 generic.go:356] "Generic (PLEG): container finished" podID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerID="1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2" exitCode=0 Oct 31 13:47:01.553820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.553619 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" Oct 31 13:47:01.553820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.553663 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" event={"ID":"febb9a1e-921f-4e7d-9af7-a31a8113fbba","Type":"ContainerDied","Data":"1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2"} Oct 31 13:47:01.553820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.553710 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n" event={"ID":"febb9a1e-921f-4e7d-9af7-a31a8113fbba","Type":"ContainerDied","Data":"9d5dc7615cb730770721c51c355888214a6cbc8a66965c82a2a7a35bb7c899d1"} Oct 31 13:47:01.553820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.553726 5074 scope.go:122] "RemoveContainer" containerID="1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2" Oct 31 13:47:01.564193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.564174 5074 scope.go:122] "RemoveContainer" containerID="1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2" Oct 31 13:47:01.564507 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:47:01.564487 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2\": container with ID starting with 1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2 not found: ID does not exist" containerID="1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2" Oct 31 13:47:01.564590 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.564518 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2"} err="failed to get container status \"1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2\": rpc error: code = NotFound desc = could not find container \"1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2\": container with ID starting with 1d821c5329999dde75439b29f1c0fe50d47fbb31ff736aff118f437d717046d2 not found: ID does not exist" Oct 31 13:47:01.577423 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.577393 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n"] Oct 31 13:47:01.581694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.581669 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ae9e7/distinct-w4-7ae9e7-kserve-869bd94887-sgq9n"] Oct 31 13:47:01.937335 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:01.937256 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" path="/var/lib/kubelet/pods/febb9a1e-921f-4e7d-9af7-a31a8113fbba/volumes" Oct 31 13:47:05.174409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:05.174366 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:47:06.038091 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:06.038052 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc"] Oct 31 13:47:06.041758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:06.041732 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc203ecb-b24jc"] Oct 31 13:47:07.937181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:07.937144 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a3b4ce-2473-4d61-a213-f212d50d7a2a" path="/var/lib/kubelet/pods/43a3b4ce-2473-4d61-a213-f212d50d7a2a/volumes" Oct 31 13:47:08.022897 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:08.022860 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6"] Oct 31 13:47:08.026259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:08.026224 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e47c272-xmvp6"] Oct 31 13:47:09.936517 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:09.936479 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece3289c-d09c-4ac5-8a62-003d291e2b14" path="/var/lib/kubelet/pods/ece3289c-d09c-4ac5-8a62-003d291e2b14/volumes" Oct 31 13:47:14.443016 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:14.442978 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r6gnb_6673fd3f-afa0-4106-901a-7afcad6cdeee/manager/0.log" Oct 31 13:47:15.289979 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:15.289934 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jsr6j_87dfb15b-fa89-461c-b36a-bceb692300c0/manager/0.log" Oct 31 13:47:15.499399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:15.499363 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6jt82_fbeeade1-58fc-47fb-b398-6547ad67129b/limitador/0.log" Oct 31 13:47:16.024715 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:16.024676 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26"] Oct 31 13:47:16.026886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:16.026858 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeac515-d7z26"] Oct 31 13:47:17.040351 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:17.040313 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tpdfx_c39defbd-ec41-4ab3-a577-ba969686c319/manager/0.log" Oct 31 13:47:17.260439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:17.260398 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xtpgb_a830a12a-e88b-4cca-97f6-8aa69759dfbd/manager/0.log" Oct 31 13:47:17.937451 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:17.937416 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b73787-fef7-41d2-8105-feb5dcf6805a" path="/var/lib/kubelet/pods/50b73787-fef7-41d2-8105-feb5dcf6805a/volumes" Oct 31 13:47:19.023944 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:19.023907 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt"] Oct 31 13:47:19.025930 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:19.025899 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd6b27f-np5nt"] Oct 31 13:47:19.936674 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:19.936621 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87d871b-7840-429c-a655-0b8be93f4616" path="/var/lib/kubelet/pods/f87d871b-7840-429c-a655-0b8be93f4616/volumes" Oct 31 13:47:25.982764 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:47:25.982724 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:47:27.037081 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037018 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2"] Oct 31 13:47:27.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037466 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2dd55587-a53c-4c0e-8642-15260308824a" containerName="main" Oct 31 13:47:27.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037480 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd55587-a53c-4c0e-8642-15260308824a" containerName="main" Oct 31 13:47:27.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037497 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7a588df7-2573-462a-b898-637c548384a1" containerName="revoke-keys" Oct 31 13:47:27.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037503 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a588df7-2573-462a-b898-637c548384a1" containerName="revoke-keys" Oct 31 13:47:27.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037513 5074 cpu_manager.go:414] "containerMap: removing container" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" Oct 31 13:47:27.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037520 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" Oct 31 13:47:27.037528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037530 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037535 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037543 5074 cpu_manager.go:414] "containerMap: removing container" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037548 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037556 5074 cpu_manager.go:414] "containerMap: removing container" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037561 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037570 5074 cpu_manager.go:414] "containerMap: removing container" podUID="703441e7-a964-49b8-8af3-8c951959bd22" containerName="payload-pre-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037578 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703441e7-a964-49b8-8af3-8c951959bd22" containerName="payload-pre-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037586 5074 cpu_manager.go:414] "containerMap: removing container" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037591 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037605 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037610 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037617 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037621 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037627 5074 cpu_manager.go:414] "containerMap: removing container" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037645 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037658 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037663 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037672 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037677 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037684 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037688 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037699 5074 cpu_manager.go:414] "containerMap: removing container" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037703 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037710 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037715 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037723 5074 cpu_manager.go:414] "containerMap: removing container" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037730 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037739 5074 cpu_manager.go:414] "containerMap: removing container" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037744 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037751 5074 cpu_manager.go:414] "containerMap: removing container" podUID="38f511f1-b31c-44aa-a6de-04741bfe6f08" containerName="payload-pre-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037755 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f511f1-b31c-44aa-a6de-04741bfe6f08" containerName="payload-pre-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037765 5074 cpu_manager.go:414] "containerMap: removing container" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037770 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037776 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" containerName="payload-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037780 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" containerName="payload-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037787 5074 cpu_manager.go:414] "containerMap: removing container" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037792 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037803 5074 cpu_manager.go:414] "containerMap: removing container" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037811 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037821 5074 cpu_manager.go:414] "containerMap: removing container" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037827 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037892 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a588df7-2573-462a-b898-637c548384a1" containerName="revoke-keys" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037904 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c03650-44c5-4c06-b8a3-940deaa36257" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037912 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94beb7c-137f-4c8c-b08e-cf95d981f725" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037918 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037923 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a85d43-3cca-48ff-b8a3-f8de2beff5b1" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037929 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="703441e7-a964-49b8-8af3-8c951959bd22" containerName="payload-pre-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037935 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c12e2a-9e24-4adf-be3c-279947b46915" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037941 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037946 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037951 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="febb9a1e-921f-4e7d-9af7-a31a8113fbba" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037959 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037967 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037979 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037987 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.037994 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ef088b-8828-4966-9457-fe4346c638b1" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038000 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038005 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a2814f-e408-4a17-9e98-39c7227ee416" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038010 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="845e6ed7-6c57-445f-acc3-8bd3f71bb72f" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038016 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f511f1-b31c-44aa-a6de-04741bfe6f08" containerName="payload-pre-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038021 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="296a60e8-1efa-44da-bed6-842ccefdd93d" containerName="main" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038027 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5529ab24-7af7-4584-94cf-c6a6f485cd3f" containerName="payload-processing" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038037 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038045 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5a45d2-e05f-4d34-9bdd-36fb89b8d36a" containerName="istio-proxy" Oct 31 13:47:27.038070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.038052 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd55587-a53c-4c0e-8642-15260308824a" containerName="main" Oct 31 13:47:27.041249 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041229 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c"] Oct 31 13:47:27.041385 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041364 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.041755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041733 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.041808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041759 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.041808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041800 5074 cpu_manager.go:414] "containerMap: removing container" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.041808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041806 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.041947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041823 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.041947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041829 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc96ebae-a48c-4fff-8192-a013c6eaaead" containerName="cleanup" Oct 31 13:47:27.041947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041900 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a55780-f5ee-4565-82ed-b4eaaa2b4ad9" containerName="cleanup" Oct 31 13:47:27.041947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041911 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa537dee-f5b5-4c47-932f-3c9ee78d9c8e" containerName="cleanup" Oct 31 13:47:27.041947 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.041918 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bda705-a0e5-4214-b53d-557561d0a67f" containerName="cleanup" Oct 31 13:47:27.044939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.044917 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.053507 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.053482 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2"] Oct 31 13:47:27.053771 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.053750 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8504d853\"" Oct 31 13:47:27.053933 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.053894 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8504d853-dockercfg-9j9gc\"" Oct 31 13:47:27.062307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.062261 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c"] Oct 31 13:47:27.067861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.067838 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk"] Oct 31 13:47:27.070119 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.070095 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab9bb5c-mxngk"] Oct 31 13:47:27.191430 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.191385 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztkh\" (UniqueName: \"kubernetes.io/projected/6945f76e-af84-4feb-94a3-5c52486c8acd-kube-api-access-nztkh\") pod \"payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.191624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.191445 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6945f76e-af84-4feb-94a3-5c52486c8acd-plugins-config-volume\") pod \"payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.191624 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.191571 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx72r\" (UniqueName: \"kubernetes.io/projected/479c0754-70a9-4aac-ad60-049aae6455ab-kube-api-access-qx72r\") pod \"payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.191733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.191715 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c0754-70a9-4aac-ad60-049aae6455ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.293226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.293137 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6945f76e-af84-4feb-94a3-5c52486c8acd-plugins-config-volume\") pod \"payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.293226 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.293201 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx72r\" (UniqueName: \"kubernetes.io/projected/479c0754-70a9-4aac-ad60-049aae6455ab-kube-api-access-qx72r\") pod \"payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.293438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.293268 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c0754-70a9-4aac-ad60-049aae6455ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.293438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.293322 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztkh\" (UniqueName: \"kubernetes.io/projected/6945f76e-af84-4feb-94a3-5c52486c8acd-kube-api-access-nztkh\") pod \"payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.294017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.293993 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6945f76e-af84-4feb-94a3-5c52486c8acd-plugins-config-volume\") pod \"payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.294069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.294006 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c0754-70a9-4aac-ad60-049aae6455ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.315834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.315805 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztkh\" (UniqueName: \"kubernetes.io/projected/6945f76e-af84-4feb-94a3-5c52486c8acd-kube-api-access-nztkh\") pod \"payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.316044 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.316026 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx72r\" (UniqueName: \"kubernetes.io/projected/479c0754-70a9-4aac-ad60-049aae6455ab-kube-api-access-qx72r\") pod \"payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.357561 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.357529 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.365581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.365549 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.647792 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.647760 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2"] Oct 31 13:47:27.649315 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:47:27.649283 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479c0754_70a9_4aac_ad60_049aae6455ab.slice/crio-d703ff52d93d1d1eb6ac39eb0df553518b7ff3ff99e704c84f5d1b209926a494 WatchSource:0}: Error finding container d703ff52d93d1d1eb6ac39eb0df553518b7ff3ff99e704c84f5d1b209926a494: Status 404 returned error can't find the container with id d703ff52d93d1d1eb6ac39eb0df553518b7ff3ff99e704c84f5d1b209926a494 Oct 31 13:47:27.660362 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.660338 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c"] Oct 31 13:47:27.663169 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:47:27.663138 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6945f76e_af84_4feb_94a3_5c52486c8acd.slice/crio-47dc36dab6bf3e3270a063b6d9777f4b5ecd62ac25f5ac4cc7c4baadf66deba6 WatchSource:0}: Error finding container 47dc36dab6bf3e3270a063b6d9777f4b5ecd62ac25f5ac4cc7c4baadf66deba6: Status 404 returned error can't find the container with id 47dc36dab6bf3e3270a063b6d9777f4b5ecd62ac25f5ac4cc7c4baadf66deba6 Oct 31 13:47:27.727536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.727497 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" event={"ID":"479c0754-70a9-4aac-ad60-049aae6455ab","Type":"ContainerStarted","Data":"362b34658bfe4944d33ac6cac8e069d24cb0c9a5726357570874a546eed9c134"} Oct 31 13:47:27.727762 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.727743 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:27.727840 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.727790 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" event={"ID":"479c0754-70a9-4aac-ad60-049aae6455ab","Type":"ContainerStarted","Data":"d703ff52d93d1d1eb6ac39eb0df553518b7ff3ff99e704c84f5d1b209926a494"} Oct 31 13:47:27.729432 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.729405 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" event={"ID":"6945f76e-af84-4feb-94a3-5c52486c8acd","Type":"ContainerStarted","Data":"cdbea7c6c9981bef0a246ee17881ae52b40b9b06f3ccb0fb25ad67e98212816c"} Oct 31 13:47:27.729534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.729455 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:27.729534 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.729477 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" event={"ID":"6945f76e-af84-4feb-94a3-5c52486c8acd","Type":"ContainerStarted","Data":"47dc36dab6bf3e3270a063b6d9777f4b5ecd62ac25f5ac4cc7c4baadf66deba6"} Oct 31 13:47:27.778966 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.778897 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" podStartSLOduration=1.778882726 podStartE2EDuration="1.778882726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:27.777505442 +0000 UTC m=+4541.955223426" watchObservedRunningTime="2026-10-31 13:47:27.778882726 +0000 UTC m=+4541.956600757" Oct 31 13:47:27.831797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.831731 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" podStartSLOduration=0.831707844 podStartE2EDuration="831.707844ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:27.828604799 +0000 UTC m=+4542.006322784" watchObservedRunningTime="2026-10-31 13:47:27.831707844 +0000 UTC m=+4542.009425829" Oct 31 13:47:27.938029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:27.937931 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a" path="/var/lib/kubelet/pods/cf2ac7bc-ce3c-4771-a315-3e2fe96ebf7a/volumes" Oct 31 13:47:29.696673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:29.696620 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:47:38.739268 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:38.739235 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:38.739815 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:38.739373 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:41.572711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.572668 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2"] Oct 31 13:47:41.577720 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.577701 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.586133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.586099 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:47:41.587238 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.587213 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:47:41.587360 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.587225 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:47:41.587441 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.587426 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfmv6\"" Oct 31 13:47:41.590950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.590927 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2"] Oct 31 13:47:41.624376 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.624341 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.624557 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.624532 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.624623 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.624588 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bsb\" (UniqueName: \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.725286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.725250 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.725490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.725303 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bsb\" (UniqueName: \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.725490 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.725346 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.726106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.726079 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.727680 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:41.727656 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:41.734850 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:47:41.734824 5074 projected.go:196] Error preparing data for projected volume kube-api-access-c8bsb for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 13:47:41.735015 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:47:41.734903 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb podName:ce815a51-8f02-45e4-82d9-13f90ef7e0b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:47:42.234882705 +0000 UTC m=+4556.412600667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c8bsb" (UniqueName: "kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb") pod "maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" (UID: "ce815a51-8f02-45e4-82d9-13f90ef7e0b6") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 13:47:42.331073 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:42.331022 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bsb\" (UniqueName: \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:42.336619 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:47:42.336595 5074 projected.go:196] Error preparing data for projected volume kube-api-access-c8bsb for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 13:47:42.336755 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:47:42.336695 5074 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb podName:ce815a51-8f02-45e4-82d9-13f90ef7e0b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:47:43.336678572 +0000 UTC m=+4557.514396538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c8bsb" (UniqueName: "kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb") pod "maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" (UID: "ce815a51-8f02-45e4-82d9-13f90ef7e0b6") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 13:47:43.341735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:43.341701 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bsb\" (UniqueName: \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:43.358821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:43.358789 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bsb\" (UniqueName: \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\") pod \"maas-api-revoke-keys-e2e-ait-8504d853-2d7x2\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:43.392361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:43.392331 5074 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" secret="" err="secret \"maas-api-cleanup-dockercfg-sfmv6\" not found" Oct 31 13:47:43.392521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:43.392377 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:43.674770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:43.674741 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2"] Oct 31 13:47:43.676369 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:47:43.676337 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce815a51_8f02_45e4_82d9_13f90ef7e0b6.slice/crio-9fb291f4c3ff515f64bf229e375961612bc23d318cfcd450919863591402f584 WatchSource:0}: Error finding container 9fb291f4c3ff515f64bf229e375961612bc23d318cfcd450919863591402f584: Status 404 returned error can't find the container with id 9fb291f4c3ff515f64bf229e375961612bc23d318cfcd450919863591402f584 Oct 31 13:47:43.826117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:43.826076 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" event={"ID":"ce815a51-8f02-45e4-82d9-13f90ef7e0b6","Type":"ContainerStarted","Data":"9fb291f4c3ff515f64bf229e375961612bc23d318cfcd450919863591402f584"} Oct 31 13:47:44.832864 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:44.832830 5074 generic.go:356] "Generic (PLEG): container finished" podID="ce815a51-8f02-45e4-82d9-13f90ef7e0b6" containerID="8165b5a3ed1ba3b29dbc9affb0090418ab6ba0919554c465aafb4025494f6641" exitCode=0 Oct 31 13:47:44.833295 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:44.832915 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" event={"ID":"ce815a51-8f02-45e4-82d9-13f90ef7e0b6","Type":"ContainerDied","Data":"8165b5a3ed1ba3b29dbc9affb0090418ab6ba0919554c465aafb4025494f6641"} Oct 31 13:47:45.974928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:45.974902 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:46.067794 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.067761 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-xks-ca\") pod \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " Oct 31 13:47:46.067967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.067822 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\" (UniqueName: \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\") pod \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " Oct 31 13:47:46.067967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.067878 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-openshift-ca\") pod \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\" (UID: \"ce815a51-8f02-45e4-82d9-13f90ef7e0b6\") " Oct 31 13:47:46.068352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.068303 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-openshift-ca" pod "ce815a51-8f02-45e4-82d9-13f90ef7e0b6" (UID: "ce815a51-8f02-45e4-82d9-13f90ef7e0b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:46.070108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.070078 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb" pod "ce815a51-8f02-45e4-82d9-13f90ef7e0b6" (UID: "ce815a51-8f02-45e4-82d9-13f90ef7e0b6"). InnerVolumeSpecName "kube-api-access-c8bsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:46.070374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.070347 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-xks-ca" pod "ce815a51-8f02-45e4-82d9-13f90ef7e0b6" (UID: "ce815a51-8f02-45e4-82d9-13f90ef7e0b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:47:46.168863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.168793 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:46.168863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.168822 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bsb\" (UniqueName: \"kubernetes.io/projected/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-kube-api-access-c8bsb\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:46.168863 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.168834 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce815a51-8f02-45e4-82d9-13f90ef7e0b6-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:46.845108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.845079 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" Oct 31 13:47:46.847498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.845074 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2" event={"ID":"ce815a51-8f02-45e4-82d9-13f90ef7e0b6","Type":"ContainerDied","Data":"9fb291f4c3ff515f64bf229e375961612bc23d318cfcd450919863591402f584"} Oct 31 13:47:46.847498 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:46.845229 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb291f4c3ff515f64bf229e375961612bc23d318cfcd450919863591402f584" Oct 31 13:47:51.704488 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:51.704446 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c"] Oct 31 13:47:51.716470 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:51.716438 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2"] Oct 31 13:47:52.029951 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:52.029907 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9"] Oct 31 13:47:52.033286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:52.033258 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d5cf47-6xzr9"] Oct 31 13:47:52.995703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:52.995666 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2"] Oct 31 13:47:52.997839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:52.997814 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8504d853-2d7x2"] Oct 31 13:47:53.937282 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:53.937244 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6783d96d-5e10-4760-9dd8-a79d495ff706" path="/var/lib/kubelet/pods/6783d96d-5e10-4760-9dd8-a79d495ff706/volumes" Oct 31 13:47:53.937703 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:53.937688 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce815a51-8f02-45e4-82d9-13f90ef7e0b6" path="/var/lib/kubelet/pods/ce815a51-8f02-45e4-82d9-13f90ef7e0b6/volumes" Oct 31 13:47:54.023139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:54.023096 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn"] Oct 31 13:47:54.025921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:54.025896 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86efb512-w6dpn"] Oct 31 13:47:54.544142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:54.544102 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:47:55.937510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:55.937463 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3560fe0a-a54a-4a6c-8df4-a47145c602ed" path="/var/lib/kubelet/pods/3560fe0a-a54a-4a6c-8df4-a47145c602ed/volumes" Oct 31 13:47:55.984384 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:47:55.984347 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:47:56.705008 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.704930 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" podUID="6945f76e-af84-4feb-94a3-5c52486c8acd" containerName="payload-processing" containerID="cri-o://cdbea7c6c9981bef0a246ee17881ae52b40b9b06f3ccb0fb25ad67e98212816c" gracePeriod=25 Oct 31 13:47:56.717666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.717604 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" podUID="479c0754-70a9-4aac-ad60-049aae6455ab" containerName="payload-pre-processing" containerID="cri-o://362b34658bfe4944d33ac6cac8e069d24cb0c9a5726357570874a546eed9c134" gracePeriod=25 Oct 31 13:47:56.736442 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.736413 5074 scope.go:122] "RemoveContainer" containerID="cfe086beb0751848fb0d7ca2cbac82f06489dfb1b6654eeac19547a908cecec4" Oct 31 13:47:56.844511 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.844118 5074 scope.go:122] "RemoveContainer" containerID="71021dac81160652d6df95a4ea5f1ab42f52c462282c6642192a1320d9383f35" Oct 31 13:47:56.910049 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.910014 5074 generic.go:356] "Generic (PLEG): container finished" podID="479c0754-70a9-4aac-ad60-049aae6455ab" containerID="362b34658bfe4944d33ac6cac8e069d24cb0c9a5726357570874a546eed9c134" exitCode=0 Oct 31 13:47:56.910232 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.910124 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" event={"ID":"479c0754-70a9-4aac-ad60-049aae6455ab","Type":"ContainerDied","Data":"362b34658bfe4944d33ac6cac8e069d24cb0c9a5726357570874a546eed9c134"} Oct 31 13:47:56.912138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.912109 5074 generic.go:356] "Generic (PLEG): container finished" podID="6945f76e-af84-4feb-94a3-5c52486c8acd" containerID="cdbea7c6c9981bef0a246ee17881ae52b40b9b06f3ccb0fb25ad67e98212816c" exitCode=0 Oct 31 13:47:56.912266 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.912196 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" event={"ID":"6945f76e-af84-4feb-94a3-5c52486c8acd","Type":"ContainerDied","Data":"cdbea7c6c9981bef0a246ee17881ae52b40b9b06f3ccb0fb25ad67e98212816c"} Oct 31 13:47:56.973219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:56.973189 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:57.025054 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.025029 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:57.072808 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.072777 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6945f76e-af84-4feb-94a3-5c52486c8acd-kube-api-access-nztkh\" (UniqueName: \"kubernetes.io/projected/6945f76e-af84-4feb-94a3-5c52486c8acd-kube-api-access-nztkh\") pod \"6945f76e-af84-4feb-94a3-5c52486c8acd\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " Oct 31 13:47:57.072975 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.072838 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6945f76e-af84-4feb-94a3-5c52486c8acd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6945f76e-af84-4feb-94a3-5c52486c8acd-plugins-config-volume\") pod \"6945f76e-af84-4feb-94a3-5c52486c8acd\" (UID: \"6945f76e-af84-4feb-94a3-5c52486c8acd\") " Oct 31 13:47:57.073183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.073160 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6945f76e-af84-4feb-94a3-5c52486c8acd-plugins-config-volume" pod "6945f76e-af84-4feb-94a3-5c52486c8acd" (UID: "6945f76e-af84-4feb-94a3-5c52486c8acd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:57.074988 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.074954 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6945f76e-af84-4feb-94a3-5c52486c8acd-kube-api-access-nztkh" pod "6945f76e-af84-4feb-94a3-5c52486c8acd" (UID: "6945f76e-af84-4feb-94a3-5c52486c8acd"). InnerVolumeSpecName "kube-api-access-nztkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:57.173798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.173764 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479c0754-70a9-4aac-ad60-049aae6455ab-kube-api-access-qx72r\" (UniqueName: \"kubernetes.io/projected/479c0754-70a9-4aac-ad60-049aae6455ab-kube-api-access-qx72r\") pod \"479c0754-70a9-4aac-ad60-049aae6455ab\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " Oct 31 13:47:57.173989 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.173966 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479c0754-70a9-4aac-ad60-049aae6455ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c0754-70a9-4aac-ad60-049aae6455ab-plugins-config-volume\") pod \"479c0754-70a9-4aac-ad60-049aae6455ab\" (UID: \"479c0754-70a9-4aac-ad60-049aae6455ab\") " Oct 31 13:47:57.174285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.174256 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztkh\" (UniqueName: \"kubernetes.io/projected/6945f76e-af84-4feb-94a3-5c52486c8acd-kube-api-access-nztkh\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:57.174285 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.174287 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6945f76e-af84-4feb-94a3-5c52486c8acd-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:57.174453 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.174364 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479c0754-70a9-4aac-ad60-049aae6455ab-plugins-config-volume" pod "479c0754-70a9-4aac-ad60-049aae6455ab" (UID: "479c0754-70a9-4aac-ad60-049aae6455ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:57.175765 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.175738 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479c0754-70a9-4aac-ad60-049aae6455ab-kube-api-access-qx72r" pod "479c0754-70a9-4aac-ad60-049aae6455ab" (UID: "479c0754-70a9-4aac-ad60-049aae6455ab"). InnerVolumeSpecName "kube-api-access-qx72r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:57.274874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.274783 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c0754-70a9-4aac-ad60-049aae6455ab-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:57.274874 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.274822 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx72r\" (UniqueName: \"kubernetes.io/projected/479c0754-70a9-4aac-ad60-049aae6455ab-kube-api-access-qx72r\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:57.919474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.919440 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" Oct 31 13:47:57.919698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.919488 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c" event={"ID":"6945f76e-af84-4feb-94a3-5c52486c8acd","Type":"ContainerDied","Data":"47dc36dab6bf3e3270a063b6d9777f4b5ecd62ac25f5ac4cc7c4baadf66deba6"} Oct 31 13:47:57.919698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.919562 5074 scope.go:122] "RemoveContainer" containerID="cdbea7c6c9981bef0a246ee17881ae52b40b9b06f3ccb0fb25ad67e98212816c" Oct 31 13:47:57.921107 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.921085 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" Oct 31 13:47:57.921186 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.921068 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2" event={"ID":"479c0754-70a9-4aac-ad60-049aae6455ab","Type":"ContainerDied","Data":"d703ff52d93d1d1eb6ac39eb0df553518b7ff3ff99e704c84f5d1b209926a494"} Oct 31 13:47:57.931541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.931519 5074 scope.go:122] "RemoveContainer" containerID="362b34658bfe4944d33ac6cac8e069d24cb0c9a5726357570874a546eed9c134" Oct 31 13:47:57.945530 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.945506 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c"] Oct 31 13:47:57.950663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.950626 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8504d853-68c9875d89-qbh9c"] Oct 31 13:47:57.961317 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.961293 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2"] Oct 31 13:47:57.966986 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:57.966962 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8504d853-84d58db7ff-nqjv2"] Oct 31 13:47:59.937003 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:59.936963 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479c0754-70a9-4aac-ad60-049aae6455ab" path="/var/lib/kubelet/pods/479c0754-70a9-4aac-ad60-049aae6455ab/volumes" Oct 31 13:47:59.937571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:47:59.937549 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6945f76e-af84-4feb-94a3-5c52486c8acd" path="/var/lib/kubelet/pods/6945f76e-af84-4feb-94a3-5c52486c8acd/volumes" Oct 31 13:48:04.023204 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:04.023164 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf"] Oct 31 13:48:04.025595 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:04.025572 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816023d3-bqtkf"] Oct 31 13:48:05.936690 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:05.936656 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b771d36c-cd7c-4421-858a-73df3e933269" path="/var/lib/kubelet/pods/b771d36c-cd7c-4421-858a-73df3e933269/volumes" Oct 31 13:48:09.023395 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:09.023345 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6"] Oct 31 13:48:09.026112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:09.026078 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95dc00-tq5f6"] Oct 31 13:48:09.937529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:09.937493 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9686267f-a2ad-4931-b2c0-eeb5e712540a" path="/var/lib/kubelet/pods/9686267f-a2ad-4931-b2c0-eeb5e712540a/volumes" Oct 31 13:48:23.740464 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.740426 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x"] Oct 31 13:48:23.741032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741007 5074 cpu_manager.go:414] "containerMap: removing container" podUID="479c0754-70a9-4aac-ad60-049aae6455ab" containerName="payload-pre-processing" Oct 31 13:48:23.741114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741036 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479c0754-70a9-4aac-ad60-049aae6455ab" containerName="payload-pre-processing" Oct 31 13:48:23.741114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741082 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6945f76e-af84-4feb-94a3-5c52486c8acd" containerName="payload-processing" Oct 31 13:48:23.741114 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741092 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6945f76e-af84-4feb-94a3-5c52486c8acd" containerName="payload-processing" Oct 31 13:48:23.741283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741129 5074 cpu_manager.go:414] "containerMap: removing container" podUID="ce815a51-8f02-45e4-82d9-13f90ef7e0b6" containerName="revoke-keys" Oct 31 13:48:23.741283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741138 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce815a51-8f02-45e4-82d9-13f90ef7e0b6" containerName="revoke-keys" Oct 31 13:48:23.741283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741236 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="479c0754-70a9-4aac-ad60-049aae6455ab" containerName="payload-pre-processing" Oct 31 13:48:23.741283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741255 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6945f76e-af84-4feb-94a3-5c52486c8acd" containerName="payload-processing" Oct 31 13:48:23.741283 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.741270 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce815a51-8f02-45e4-82d9-13f90ef7e0b6" containerName="revoke-keys" Oct 31 13:48:23.746410 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.746386 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.773004 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.772978 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x"] Oct 31 13:48:23.830217 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830170 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830240 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89wl\" (UniqueName: \"kubernetes.io/projected/89558c79-c272-4647-a36f-8130cb513976-kube-api-access-v89wl\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830399 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830283 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89558c79-c272-4647-a36f-8130cb513976-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830389 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830532 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830480 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89558c79-c272-4647-a36f-8130cb513976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830542 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89558c79-c272-4647-a36f-8130cb513976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830669 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830619 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830756 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830716 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.830802 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.830770 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931267 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89558c79-c272-4647-a36f-8130cb513976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931306 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931310 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89558c79-c272-4647-a36f-8130cb513976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931419 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931450 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931489 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931541 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931535 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931565 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89wl\" (UniqueName: \"kubernetes.io/projected/89558c79-c272-4647-a36f-8130cb513976-kube-api-access-v89wl\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931603 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89558c79-c272-4647-a36f-8130cb513976-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.931814 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931659 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.932019 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.931990 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.932082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.932044 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.932152 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.932127 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.932403 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.932376 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.932797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.932772 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89558c79-c272-4647-a36f-8130cb513976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.933928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.933900 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89558c79-c272-4647-a36f-8130cb513976-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.934247 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.934228 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89558c79-c272-4647-a36f-8130cb513976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.945096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.945068 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89wl\" (UniqueName: \"kubernetes.io/projected/89558c79-c272-4647-a36f-8130cb513976-kube-api-access-v89wl\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:23.945248 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:23.945231 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89558c79-c272-4647-a36f-8130cb513976-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-szf5x\" (UID: \"89558c79-c272-4647-a36f-8130cb513976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:24.058950 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:24.058861 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:24.345117 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:24.345087 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x"] Oct 31 13:48:24.347393 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:48:24.347366 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89558c79_c272_4647_a36f_8130cb513976.slice/crio-55584829bd00e274949849a3989fd6e5e79cb01cc2d43cd9390129c98a4f6121 WatchSource:0}: Error finding container 55584829bd00e274949849a3989fd6e5e79cb01cc2d43cd9390129c98a4f6121: Status 404 returned error can't find the container with id 55584829bd00e274949849a3989fd6e5e79cb01cc2d43cd9390129c98a4f6121 Oct 31 13:48:24.350078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:24.350048 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:24.350156 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:24.350115 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:24.350156 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:24.350146 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:25.073664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:25.073229 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" event={"ID":"89558c79-c272-4647-a36f-8130cb513976","Type":"ContainerStarted","Data":"7992c498bf568f7e82eec55c1401e9804755928f939108f40a3f20ec600ff791"} Oct 31 13:48:25.073664 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:25.073301 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" event={"ID":"89558c79-c272-4647-a36f-8130cb513976","Type":"ContainerStarted","Data":"55584829bd00e274949849a3989fd6e5e79cb01cc2d43cd9390129c98a4f6121"} Oct 31 13:48:25.103742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:25.103674 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podStartSLOduration=2.103654113 podStartE2EDuration="2.103654113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:25.100265958 +0000 UTC m=+4599.277983942" watchObservedRunningTime="2026-10-31 13:48:25.103654113 +0000 UTC m=+4599.281372098" Oct 31 13:48:26.011068 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:48:26.011028 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:48:26.059214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:26.059181 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:26.060932 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:26.060899 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Oct 31 13:48:26.061066 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:26.060956 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Oct 31 13:48:27.059953 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:27.059916 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Oct 31 13:48:27.060367 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:27.059973 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Oct 31 13:48:28.059742 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:28.059701 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Oct 31 13:48:28.059936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:28.059766 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Oct 31 13:48:29.059472 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:29.059423 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Oct 31 13:48:29.060042 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:29.059484 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Oct 31 13:48:30.059328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:30.059275 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Oct 31 13:48:30.059510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:30.059339 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Oct 31 13:48:31.060305 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:31.060265 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Oct 31 13:48:31.060861 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:31.060331 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Oct 31 13:48:32.678713 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:32.678667 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:32.679214 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:32.679102 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:32.680836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:32.680809 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" Oct 31 13:48:55.981564 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:48:55.981518 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:48:56.898420 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.898381 5074 scope.go:122] "RemoveContainer" containerID="29377f311874b25278c70d95ffb218eef59a028c9ed523021fc6a1168f1a5910" Oct 31 13:48:56.907671 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.907646 5074 scope.go:122] "RemoveContainer" containerID="36d3eca70e139d77f7ca4ab8d45b2bd5362a17b5a9e5529eb3fd33de765b2115" Oct 31 13:48:56.917796 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.917769 5074 scope.go:122] "RemoveContainer" containerID="bfbc5f9a2b8fceba47cfb3c3440dbe8f3fc0d9210de9fbd1e734ef94a407e8aa" Oct 31 13:48:56.926096 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.926071 5074 scope.go:122] "RemoveContainer" containerID="a9b54e83239085db556af16033d552698134cbf6ba2ca0d59e771f390ddab6de" Oct 31 13:48:56.934113 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.934093 5074 scope.go:122] "RemoveContainer" containerID="9bcc20205c241f783a8757dfac23ad150f553b4332aa3a2ff1021a76e09bb82b" Oct 31 13:48:56.942071 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.942054 5074 scope.go:122] "RemoveContainer" containerID="cacc2ba08dd0a44b8857ce9707ed98ce92f857b7bb39866f4a5b7a5405db7713" Oct 31 13:48:56.950935 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.950917 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:48:56.950997 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:48:56.950982 5074 scope.go:122] "RemoveContainer" containerID="2684f13550f75bec98185b432648ccb9ae85fca2ce16021964cb59a2e7e3730b" Oct 31 13:49:12.446393 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:12.446356 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:49:17.029392 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:17.029357 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs"] Oct 31 13:49:17.042596 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:17.042564 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68bc77b-cdccs"] Oct 31 13:49:17.396556 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:17.396467 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:49:17.937122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:17.937084 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25de364-7b78-4e72-b4cb-ca3787b29fe1" path="/var/lib/kubelet/pods/f25de364-7b78-4e72-b4cb-ca3787b29fe1/volumes" Oct 31 13:49:20.443120 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:20.443083 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:49:25.981988 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:49:25.981939 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:49:29.023967 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:29.023934 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862"] Oct 31 13:49:29.027734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:29.027709 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50523673-dz862"] Oct 31 13:49:29.937512 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:29.937475 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77aff0fd-d5b7-4943-bcac-2a59bd0a66fc" path="/var/lib/kubelet/pods/77aff0fd-d5b7-4943-bcac-2a59bd0a66fc/volumes" Oct 31 13:49:35.574620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:35.574582 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:49:40.218093 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:40.218051 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:49:43.143987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:43.143946 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:49:50.022901 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:50.022862 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff"] Oct 31 13:49:50.025377 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:50.025351 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a5aebf-bjzff"] Oct 31 13:49:51.937034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:51.936997 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5ec8be-e55f-4517-a5d1-03c283bc2c69" path="/var/lib/kubelet/pods/ac5ec8be-e55f-4517-a5d1-03c283bc2c69/volumes" Oct 31 13:49:55.981420 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:49:55.981387 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:49:57.005999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:57.005958 5074 scope.go:122] "RemoveContainer" containerID="7970aae06c487e6bbeea814effc79e7e40c5683ba353c514a1248932ffa67ba8" Oct 31 13:49:57.015577 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:57.015559 5074 scope.go:122] "RemoveContainer" containerID="00d6f0e2608638c045df03c16559b6be3fe447f620bc355b14de9e6f4d974ddb" Oct 31 13:49:57.023821 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:57.023801 5074 scope.go:122] "RemoveContainer" containerID="4fdc0272a3684cb2a7045222a6ebdb5b5a3e2074bee6622c21b65fc7386d0ee7" Oct 31 13:49:58.028103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:58.028063 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh"] Oct 31 13:49:58.032689 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:58.032662 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826c638f-cfrdh"] Oct 31 13:49:59.936849 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:49:59.936809 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c98812-e834-4f7e-91f4-41a22574c8e4" path="/var/lib/kubelet/pods/b7c98812-e834-4f7e-91f4-41a22574c8e4/volumes" Oct 31 13:50:01.022913 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:01.022871 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59"] Oct 31 13:50:01.027728 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:01.027695 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e919951f-djv59"] Oct 31 13:50:01.938646 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:01.938596 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7842b9dc-893b-4c00-90b2-8283e5b763b3" path="/var/lib/kubelet/pods/7842b9dc-893b-4c00-90b2-8283e5b763b3/volumes" Oct 31 13:50:02.430230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:02.430189 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:50:08.512657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:08.512239 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:50:11.035144 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:11.035101 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4"] Oct 31 13:50:11.038123 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:11.038096 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbf77dd-4kll4"] Oct 31 13:50:11.937051 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:11.937017 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a588df7-2573-462a-b898-637c548384a1" path="/var/lib/kubelet/pods/7a588df7-2573-462a-b898-637c548384a1/volumes" Oct 31 13:50:25.982312 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:50:25.982273 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:50:46.441799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:46.441755 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:50:54.288770 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.288730 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk"] Oct 31 13:50:54.292548 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.292528 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.297459 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.297429 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:50:54.298586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.298564 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:50:54.298586 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.298579 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:50:54.298733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.298579 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:50:54.302748 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.302695 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk"] Oct 31 13:50:54.410893 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.410857 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4h5\" (UniqueName: \"kubernetes.io/projected/6cb73cf2-54df-4a15-afe7-33b14cf5403d-kube-api-access-xd4h5\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.411063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.410901 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.411063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.411002 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.511879 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.511841 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4h5\" (UniqueName: \"kubernetes.io/projected/6cb73cf2-54df-4a15-afe7-33b14cf5403d-kube-api-access-xd4h5\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.512053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.511888 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.512053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.511934 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.512732 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.512709 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.514353 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.514333 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.524593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.524568 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4h5\" (UniqueName: \"kubernetes.io/projected/6cb73cf2-54df-4a15-afe7-33b14cf5403d-kube-api-access-xd4h5\") pod \"maas-api-revoke-sub-bdd1d38c-fn5rk\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.607418 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.607339 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:54.881163 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:54.881090 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk"] Oct 31 13:50:54.884568 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:50:54.884538 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb73cf2_54df_4a15_afe7_33b14cf5403d.slice/crio-334f7fc994566f9dffd112aae5c352d548cc0bc3e159a240545120e470fa6078 WatchSource:0}: Error finding container 334f7fc994566f9dffd112aae5c352d548cc0bc3e159a240545120e470fa6078: Status 404 returned error can't find the container with id 334f7fc994566f9dffd112aae5c352d548cc0bc3e159a240545120e470fa6078 Oct 31 13:50:55.066510 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:55.066468 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" event={"ID":"6cb73cf2-54df-4a15-afe7-33b14cf5403d","Type":"ContainerStarted","Data":"334f7fc994566f9dffd112aae5c352d548cc0bc3e159a240545120e470fa6078"} Oct 31 13:50:55.981713 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:50:55.981668 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:50:56.073438 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:56.073400 5074 generic.go:356] "Generic (PLEG): container finished" podID="6cb73cf2-54df-4a15-afe7-33b14cf5403d" containerID="01241694894cf3b51cc9f010648ca842db87d991d101cf0a65c3d81ea8c7b276" exitCode=0 Oct 31 13:50:56.073617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:56.073485 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" event={"ID":"6cb73cf2-54df-4a15-afe7-33b14cf5403d","Type":"ContainerDied","Data":"01241694894cf3b51cc9f010648ca842db87d991d101cf0a65c3d81ea8c7b276"} Oct 31 13:50:57.051055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.051022 5074 scope.go:122] "RemoveContainer" containerID="82d304c4323067b910dcbcd5a7c637b1ed33230ef209599634a760e239b4b0f2" Oct 31 13:50:57.063869 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.063847 5074 scope.go:122] "RemoveContainer" containerID="9f9dea499bff455cec53ffaee4ad5e1728bf7f528b77921af3d622caebb2ad05" Oct 31 13:50:57.076233 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.076202 5074 scope.go:122] "RemoveContainer" containerID="f7681e6327bbf8980594ab00f26ca91fab7062cb33a07f72223cbbbc93873245" Oct 31 13:50:57.087661 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.087623 5074 scope.go:122] "RemoveContainer" containerID="443b8e71a585df6b4d9fd6d03dc18184829650b6e2d92ece8248530c97813134" Oct 31 13:50:57.225545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.225519 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:57.339993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.339890 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb73cf2-54df-4a15-afe7-33b14cf5403d-kube-api-access-xd4h5\" (UniqueName: \"kubernetes.io/projected/6cb73cf2-54df-4a15-afe7-33b14cf5403d-kube-api-access-xd4h5\") pod \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " Oct 31 13:50:57.339993 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.339954 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-openshift-ca\") pod \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " Oct 31 13:50:57.340223 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.340038 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-xks-ca\") pod \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\" (UID: \"6cb73cf2-54df-4a15-afe7-33b14cf5403d\") " Oct 31 13:50:57.340467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.340434 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-openshift-ca" pod "6cb73cf2-54df-4a15-afe7-33b14cf5403d" (UID: "6cb73cf2-54df-4a15-afe7-33b14cf5403d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:50:57.342211 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.342181 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-xks-ca" pod "6cb73cf2-54df-4a15-afe7-33b14cf5403d" (UID: "6cb73cf2-54df-4a15-afe7-33b14cf5403d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:50:57.342300 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.342210 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb73cf2-54df-4a15-afe7-33b14cf5403d-kube-api-access-xd4h5" pod "6cb73cf2-54df-4a15-afe7-33b14cf5403d" (UID: "6cb73cf2-54df-4a15-afe7-33b14cf5403d"). InnerVolumeSpecName "kube-api-access-xd4h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:50:57.441758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.441708 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:57.441758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.441753 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd4h5\" (UniqueName: \"kubernetes.io/projected/6cb73cf2-54df-4a15-afe7-33b14cf5403d-kube-api-access-xd4h5\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:57.441758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:57.441765 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb73cf2-54df-4a15-afe7-33b14cf5403d-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:58.084818 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:58.084783 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" Oct 31 13:50:58.085274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:58.084826 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk" event={"ID":"6cb73cf2-54df-4a15-afe7-33b14cf5403d","Type":"ContainerDied","Data":"334f7fc994566f9dffd112aae5c352d548cc0bc3e159a240545120e470fa6078"} Oct 31 13:50:58.085274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:50:58.084877 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="334f7fc994566f9dffd112aae5c352d548cc0bc3e159a240545120e470fa6078" Oct 31 13:51:05.288048 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:05.288009 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:51:09.663108 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.663065 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:51:09.782521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.782478 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p"] Oct 31 13:51:09.783139 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.783114 5074 cpu_manager.go:414] "containerMap: removing container" podUID="6cb73cf2-54df-4a15-afe7-33b14cf5403d" containerName="revoke-keys" Oct 31 13:51:09.783208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.783144 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb73cf2-54df-4a15-afe7-33b14cf5403d" containerName="revoke-keys" Oct 31 13:51:09.783261 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.783249 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb73cf2-54df-4a15-afe7-33b14cf5403d" containerName="revoke-keys" Oct 31 13:51:09.787100 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.787071 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.792870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.792826 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:51:09.795078 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.795051 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:51:09.795288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.795051 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:51:09.795288 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.795090 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:51:09.798263 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.798234 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p"] Oct 31 13:51:09.890059 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.890023 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.890253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.890071 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.890253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.890206 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgq8\" (UniqueName: \"kubernetes.io/projected/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-kube-api-access-rxgq8\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.991559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.991516 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.991798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.991572 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.991798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.991604 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgq8\" (UniqueName: \"kubernetes.io/projected/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-kube-api-access-rxgq8\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.992291 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.992263 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:09.994047 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:09.994018 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:10.004374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:10.004343 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgq8\" (UniqueName: \"kubernetes.io/projected/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-kube-api-access-rxgq8\") pod \"maas-api-revoke-sub-7d15814f-xz78p\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:10.103875 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:10.103820 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:10.406528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:10.406488 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p"] Oct 31 13:51:11.177050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:11.177005 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" event={"ID":"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7","Type":"ContainerStarted","Data":"10e7baabea4aad54dcf342eef7333bb09ef35b3983f6cebe54b266cfa5a906fd"} Oct 31 13:51:11.177714 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:11.177684 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" event={"ID":"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7","Type":"ContainerStarted","Data":"b80c70c43d1fa139ed49459c4cba485853fec2a154710f27f9343ac0201a9f39"} Oct 31 13:51:11.215784 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:11.215704 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" podStartSLOduration=2.215684987 podStartE2EDuration="2.215684987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:11.214144248 +0000 UTC m=+4765.391862232" watchObservedRunningTime="2026-10-31 13:51:11.215684987 +0000 UTC m=+4765.393402971" Oct 31 13:51:12.209000 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:12.208945 5074 generic.go:356] "Generic (PLEG): container finished" podID="7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" containerID="10e7baabea4aad54dcf342eef7333bb09ef35b3983f6cebe54b266cfa5a906fd" exitCode=0 Oct 31 13:51:12.209820 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:12.209745 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" event={"ID":"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7","Type":"ContainerDied","Data":"10e7baabea4aad54dcf342eef7333bb09ef35b3983f6cebe54b266cfa5a906fd"} Oct 31 13:51:12.425862 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:12.425782 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:51:12.426259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:12.426213 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:51:13.357477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.357449 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:13.458445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.458408 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-kube-api-access-rxgq8\" (UniqueName: \"kubernetes.io/projected/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-kube-api-access-rxgq8\") pod \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " Oct 31 13:51:13.458618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.458487 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-xks-ca\") pod \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " Oct 31 13:51:13.458618 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.458548 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-openshift-ca\") pod \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\" (UID: \"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7\") " Oct 31 13:51:13.458962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.458934 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-openshift-ca" pod "7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" (UID: "7f0ae20c-bd2b-4f78-95f4-8be35fe619c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:51:13.460478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.460455 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-kube-api-access-rxgq8" pod "7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" (UID: "7f0ae20c-bd2b-4f78-95f4-8be35fe619c7"). InnerVolumeSpecName "kube-api-access-rxgq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:51:13.460593 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.460561 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-xks-ca" pod "7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" (UID: "7f0ae20c-bd2b-4f78-95f4-8be35fe619c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:51:13.559918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.559881 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgq8\" (UniqueName: \"kubernetes.io/projected/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-kube-api-access-rxgq8\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:13.559918 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.559918 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:13.560133 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:13.559933 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:14.224168 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:14.224130 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" Oct 31 13:51:14.224389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:14.224141 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p" event={"ID":"7f0ae20c-bd2b-4f78-95f4-8be35fe619c7","Type":"ContainerDied","Data":"b80c70c43d1fa139ed49459c4cba485853fec2a154710f27f9343ac0201a9f39"} Oct 31 13:51:14.224389 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:14.224267 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80c70c43d1fa139ed49459c4cba485853fec2a154710f27f9343ac0201a9f39" Oct 31 13:51:21.912960 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:21.912914 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z"] Oct 31 13:51:21.913551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:21.913486 5074 cpu_manager.go:414] "containerMap: removing container" podUID="7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" containerName="revoke-keys" Oct 31 13:51:21.913551 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:21.913501 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" containerName="revoke-keys" Oct 31 13:51:21.913666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:21.913645 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" containerName="revoke-keys" Oct 31 13:51:21.917448 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:21.917420 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:21.924032 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:21.923999 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3a5889-openshift-default-dockercfg-97vbn\"" Oct 31 13:51:21.948131 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:21.948097 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z"] Oct 31 13:51:22.043296 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.043242 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-istio-token\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.043764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.043737 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e76-d039-40bc-943b-99d3778bd071-istiod-ca-cert\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.043987 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.043967 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-credential-socket\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.044145 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.044128 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-socket\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.044281 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.044257 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-envoy\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.044531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.044508 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-data\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.044668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.044546 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-certs\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.044668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.044577 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cts2m\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-kube-api-access-cts2m\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.044668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.044606 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf502e76-d039-40bc-943b-99d3778bd071-istio-podinfo\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.145734 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.145677 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-data\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.145963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.145744 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-certs\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.145963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.145782 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cts2m\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-kube-api-access-cts2m\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.145963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.145810 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf502e76-d039-40bc-943b-99d3778bd071-istio-podinfo\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.145963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.145869 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-istio-token\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.145963 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.145927 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e76-d039-40bc-943b-99d3778bd071-istiod-ca-cert\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.146253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.145982 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-credential-socket\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.146253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.146029 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-socket\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.146253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.146062 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-envoy\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.146253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.146189 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-data\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.146253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.146226 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-certs\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.146529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.146499 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-credential-socket\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.146589 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.146506 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-socket\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.147429 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.147404 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e76-d039-40bc-943b-99d3778bd071-istiod-ca-cert\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.148825 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.148801 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-envoy\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.148939 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.148849 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf502e76-d039-40bc-943b-99d3778bd071-istio-podinfo\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.159688 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.159607 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-istio-token\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.160321 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.160294 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cts2m\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-kube-api-access-cts2m\") pod \"e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.235549 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.235497 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:22.527307 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.527277 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z"] Oct 31 13:51:22.530009 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:51:22.529975 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf502e76_d039_40bc_943b_99d3778bd071.slice/crio-9ba8d3fba428ac586713c3086f45147b54f59a81df9d32bef43386a7d8bd06c3 WatchSource:0}: Error finding container 9ba8d3fba428ac586713c3086f45147b54f59a81df9d32bef43386a7d8bd06c3: Status 404 returned error can't find the container with id 9ba8d3fba428ac586713c3086f45147b54f59a81df9d32bef43386a7d8bd06c3 Oct 31 13:51:22.532583 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.532544 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:51:22.532694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.532675 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:51:22.532735 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:22.532724 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:51:23.282309 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:23.282268 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" event={"ID":"bf502e76-d039-40bc-943b-99d3778bd071","Type":"ContainerStarted","Data":"43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d"} Oct 31 13:51:23.282870 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:23.282325 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" event={"ID":"bf502e76-d039-40bc-943b-99d3778bd071","Type":"ContainerStarted","Data":"9ba8d3fba428ac586713c3086f45147b54f59a81df9d32bef43386a7d8bd06c3"} Oct 31 13:51:23.308343 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:23.308266 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podStartSLOduration=2.3082457 podStartE2EDuration="2.3082457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:23.306149108 +0000 UTC m=+4777.483867093" watchObservedRunningTime="2026-10-31 13:51:23.3082457 +0000 UTC m=+4777.485963682" Oct 31 13:51:24.236085 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:24.236040 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:24.241620 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:24.241588 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:24.289183 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:24.289146 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:24.290094 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:24.290069 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:25.983938 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:51:25.983892 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:51:26.715925 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.715882 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr"] Oct 31 13:51:26.720694 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.720668 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.738698 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.736494 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:51:26.744683 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.743972 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:51:26.745797 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.745769 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3a5889-serving-cert\"" Oct 31 13:51:26.745985 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.745838 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 13:51:26.746057 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.745902 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 13:51:26.746454 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.746434 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr"] Oct 31 13:51:26.795921 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.795858 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3774918-f6b4-42be-8f64-7b17743197fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.796121 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.796063 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.796172 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.796133 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.796210 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.796165 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqh9\" (UniqueName: \"kubernetes.io/projected/b3774918-f6b4-42be-8f64-7b17743197fa-kube-api-access-7sqh9\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.839374 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.839333 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt"] Oct 31 13:51:26.843075 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.843045 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:26.851846 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.851787 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt"] Oct 31 13:51:26.852764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.852099 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5be48be7-dockercfg-w5g5n\"" Oct 31 13:51:26.852764 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.852208 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3a5889\"" Oct 31 13:51:26.890400 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.890364 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn"] Oct 31 13:51:26.893958 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.893937 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:26.897437 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.897409 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3774918-f6b4-42be-8f64-7b17743197fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.897580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.897486 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.897662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.897619 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.897729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.897677 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqh9\" (UniqueName: \"kubernetes.io/projected/b3774918-f6b4-42be-8f64-7b17743197fa-kube-api-access-7sqh9\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.897798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.897755 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9586f702-cefc-4885-9a58-6f6a31cdde4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:26.897798 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.897791 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltpp\" (UniqueName: \"kubernetes.io/projected/9586f702-cefc-4885-9a58-6f6a31cdde4f-kube-api-access-tltpp\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:26.898355 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.898325 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3774918-f6b4-42be-8f64-7b17743197fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.900028 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.900005 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.900138 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.900107 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:26.905046 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.905018 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn"] Oct 31 13:51:26.910310 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.910261 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqh9\" (UniqueName: \"kubernetes.io/projected/b3774918-f6b4-42be-8f64-7b17743197fa-kube-api-access-7sqh9\") pod \"maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:27.000699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.998667 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9586f702-cefc-4885-9a58-6f6a31cdde4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:27.000699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.998731 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltpp\" (UniqueName: \"kubernetes.io/projected/9586f702-cefc-4885-9a58-6f6a31cdde4f-kube-api-access-tltpp\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:27.000699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.998800 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5cx\" (UniqueName: \"kubernetes.io/projected/13b95247-3a70-48dd-8984-d20c92b6744e-kube-api-access-hx5cx\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:27.000699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:26.998974 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b95247-3a70-48dd-8984-d20c92b6744e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:27.000699 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.000371 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9586f702-cefc-4885-9a58-6f6a31cdde4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:27.012505 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.012467 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltpp\" (UniqueName: \"kubernetes.io/projected/9586f702-cefc-4885-9a58-6f6a31cdde4f-kube-api-access-tltpp\") pod \"payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:27.050192 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.050149 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:27.100010 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.099959 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5cx\" (UniqueName: \"kubernetes.io/projected/13b95247-3a70-48dd-8984-d20c92b6744e-kube-api-access-hx5cx\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:27.100219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.100071 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b95247-3a70-48dd-8984-d20c92b6744e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:27.101039 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.100998 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b95247-3a70-48dd-8984-d20c92b6744e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:27.119218 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.119181 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5cx\" (UniqueName: \"kubernetes.io/projected/13b95247-3a70-48dd-8984-d20c92b6744e-kube-api-access-hx5cx\") pod \"payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:27.158905 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.158866 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:27.234717 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.234680 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:27.405407 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.405379 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr"] Oct 31 13:51:27.407154 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:51:27.407126 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3774918_f6b4_42be_8f64_7b17743197fa.slice/crio-f3ef7811a4818e056701405156f43e7afefb41b03bba802f9b8dcaaa17c93ea5 WatchSource:0}: Error finding container f3ef7811a4818e056701405156f43e7afefb41b03bba802f9b8dcaaa17c93ea5: Status 404 returned error can't find the container with id f3ef7811a4818e056701405156f43e7afefb41b03bba802f9b8dcaaa17c93ea5 Oct 31 13:51:27.408608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.408590 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:51:27.455112 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.455085 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt"] Oct 31 13:51:27.456886 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:51:27.456854 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9586f702_cefc_4885_9a58_6f6a31cdde4f.slice/crio-c4783c6b004200f0c828221765d65573810df911c278fcb56d21c58758427a4e WatchSource:0}: Error finding container c4783c6b004200f0c828221765d65573810df911c278fcb56d21c58758427a4e: Status 404 returned error can't find the container with id c4783c6b004200f0c828221765d65573810df911c278fcb56d21c58758427a4e Oct 31 13:51:27.523746 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:27.523716 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn"] Oct 31 13:51:27.525457 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:51:27.525422 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b95247_3a70_48dd_8984_d20c92b6744e.slice/crio-1ddb6ff2cafa93b72d3ed3b10a4b50f871875b5fd1133cf5c042dcc33bc5f048 WatchSource:0}: Error finding container 1ddb6ff2cafa93b72d3ed3b10a4b50f871875b5fd1133cf5c042dcc33bc5f048: Status 404 returned error can't find the container with id 1ddb6ff2cafa93b72d3ed3b10a4b50f871875b5fd1133cf5c042dcc33bc5f048 Oct 31 13:51:28.311508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.311401 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" event={"ID":"9586f702-cefc-4885-9a58-6f6a31cdde4f","Type":"ContainerStarted","Data":"f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558"} Oct 31 13:51:28.311508 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.311465 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" event={"ID":"9586f702-cefc-4885-9a58-6f6a31cdde4f","Type":"ContainerStarted","Data":"c4783c6b004200f0c828221765d65573810df911c278fcb56d21c58758427a4e"} Oct 31 13:51:28.312103 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.311600 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:28.313361 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.313327 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" event={"ID":"b3774918-f6b4-42be-8f64-7b17743197fa","Type":"ContainerStarted","Data":"322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d"} Oct 31 13:51:28.313506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.313379 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" event={"ID":"b3774918-f6b4-42be-8f64-7b17743197fa","Type":"ContainerStarted","Data":"f3ef7811a4818e056701405156f43e7afefb41b03bba802f9b8dcaaa17c93ea5"} Oct 31 13:51:28.313506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.313411 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:28.315122 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.315098 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" event={"ID":"13b95247-3a70-48dd-8984-d20c92b6744e","Type":"ContainerStarted","Data":"2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65"} Oct 31 13:51:28.315228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.315143 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" event={"ID":"13b95247-3a70-48dd-8984-d20c92b6744e","Type":"ContainerStarted","Data":"1ddb6ff2cafa93b72d3ed3b10a4b50f871875b5fd1133cf5c042dcc33bc5f048"} Oct 31 13:51:28.315228 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.315178 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:28.333128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.333074 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" podStartSLOduration=2.333061745 podStartE2EDuration="2.333061745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:28.331913286 +0000 UTC m=+4782.509631270" watchObservedRunningTime="2026-10-31 13:51:28.333061745 +0000 UTC m=+4782.510779733" Oct 31 13:51:28.351055 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.350994 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" podStartSLOduration=2.35097998 podStartE2EDuration="2.35097998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:28.349503206 +0000 UTC m=+4782.527221199" watchObservedRunningTime="2026-10-31 13:51:28.35097998 +0000 UTC m=+4782.528697960" Oct 31 13:51:28.371031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:28.370971 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" podStartSLOduration=1.792072304 podStartE2EDuration="2.370954884s" totalImagesPullingTime="578.88258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:28.370603661 +0000 UTC m=+4782.548321648" watchObservedRunningTime="2026-10-31 13:51:28.370954884 +0000 UTC m=+4782.548672867" Oct 31 13:51:29.387349 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:29.387311 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:51:34.330023 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:34.329975 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:51:39.291031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.290991 5074 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 13:51:39.291494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.291054 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 13:51:39.329449 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.329400 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:51:39.330089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.330060 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:51:39.376954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.376921 5074 generic.go:356] "Generic (PLEG): container finished" podID="bf502e76-d039-40bc-943b-99d3778bd071" containerID="43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d" exitCode=0 Oct 31 13:51:39.377142 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.376995 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" event={"ID":"bf502e76-d039-40bc-943b-99d3778bd071","Type":"ContainerDied","Data":"43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d"} Oct 31 13:51:39.377409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.377394 5074 scope.go:122] "RemoveContainer" containerID="43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d" Oct 31 13:51:39.378608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.378582 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:51:39.378687 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.378665 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:51:39.378729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:39.378720 5074 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:51:40.401258 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:40.401214 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" event={"ID":"bf502e76-d039-40bc-943b-99d3778bd071","Type":"ContainerStarted","Data":"6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a"} Oct 31 13:51:41.236799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:41.236755 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:41.237024 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:41.236998 5074 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 13:51:41.237080 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:41.237050 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 13:51:42.236686 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:42.236615 5074 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 13:51:42.237128 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:42.236705 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 13:51:43.237020 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.236982 5074 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 13:51:43.237562 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.237046 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 13:51:43.801269 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.801231 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5"] Oct 31 13:51:43.805256 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.805231 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.815880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.813508 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3a5889\"/\"openshift-service-ca.crt\"" Oct 31 13:51:43.815880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.813538 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3a5889\"/\"kube-root-ca.crt\"" Oct 31 13:51:43.815880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.813599 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3a5889\"/\"simulator-single-main-3a5889-kserve-self-signed-certs\"" Oct 31 13:51:43.816132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.816115 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3a5889\"/\"default-dockercfg-jr6q5\"" Oct 31 13:51:43.817889 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.817866 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5"] Oct 31 13:51:43.833804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.833120 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpm6w\" (UniqueName: \"kubernetes.io/projected/5fa01446-4129-4cb7-9b37-75b2ab1a708a-kube-api-access-mpm6w\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.833804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.833179 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-dshm\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.833804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.833213 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tls-certs\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.833804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.833234 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-model-cache\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.833804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.833248 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tmp-dir\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.833804 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.833263 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-home\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.934668 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.934595 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-dshm\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.934880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.934687 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tls-certs\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.934880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.934723 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-model-cache\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.934880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.934742 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tmp-dir\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.934880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.934759 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-home\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.934880 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.934839 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpm6w\" (UniqueName: \"kubernetes.io/projected/5fa01446-4129-4cb7-9b37-75b2ab1a708a-kube-api-access-mpm6w\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.935175 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.935147 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tmp-dir\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.935293 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.935265 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-home\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.935848 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.935817 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-model-cache\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.937219 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.937198 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-dshm\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.937684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.937660 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tls-certs\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:43.950208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:43.950167 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpm6w\" (UniqueName: \"kubernetes.io/projected/5fa01446-4129-4cb7-9b37-75b2ab1a708a-kube-api-access-mpm6w\") pod \"simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:44.132477 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:44.132376 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:44.236598 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:44.236552 5074 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 13:51:44.236810 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:44.236617 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 13:51:44.443789 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:44.443751 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5"] Oct 31 13:51:44.454355 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:51:44.454322 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fa01446_4129_4cb7_9b37_75b2ab1a708a.slice/crio-0b421580b29edcec0e19f83e33d34756d04a6c87198038ccba712434eb38eebf WatchSource:0}: Error finding container 0b421580b29edcec0e19f83e33d34756d04a6c87198038ccba712434eb38eebf: Status 404 returned error can't find the container with id 0b421580b29edcec0e19f83e33d34756d04a6c87198038ccba712434eb38eebf Oct 31 13:51:45.236710 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:45.236670 5074 patch_prober.go:29] interesting pod/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 13:51:45.236904 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:45.236741 5074 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 13:51:45.437202 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:45.437141 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" event={"ID":"5fa01446-4129-4cb7-9b37-75b2ab1a708a","Type":"ContainerStarted","Data":"09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43"} Oct 31 13:51:45.437415 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:45.437211 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" event={"ID":"5fa01446-4129-4cb7-9b37-75b2ab1a708a","Type":"ContainerStarted","Data":"0b421580b29edcec0e19f83e33d34756d04a6c87198038ccba712434eb38eebf"} Oct 31 13:51:45.457691 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:45.457613 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" podStartSLOduration=2.457600102 podStartE2EDuration="2.457600102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:45.457236903 +0000 UTC m=+4799.634954888" watchObservedRunningTime="2026-10-31 13:51:45.457600102 +0000 UTC m=+4799.635318085" Oct 31 13:51:45.886452 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:45.886418 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:51:46.132941 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:46.132854 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:46.240427 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:46.240396 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:46.240828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:46.240801 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:46.241592 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:46.241569 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:51:47.268617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:47.268590 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:51:47.300171 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:47.300145 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:51:54.132888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:54.132847 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:54.145124 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:54.145096 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:54.500528 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:54.500496 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:51:55.983525 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:51:55.983486 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:51:57.124484 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:57.124451 5074 scope.go:122] "RemoveContainer" containerID="0d9dd7bc3b3e22c43d968a5439e38985ed5676f9d6045fee207af4d3877f16b9" Oct 31 13:51:57.134617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:51:57.134592 5074 scope.go:122] "RemoveContainer" containerID="49b86c887b8074a7e83dd62a5efc6cd9a1207ade87cb11c640fa095e16679079" Oct 31 13:52:01.371441 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:01.371396 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:52:23.967329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:23.967291 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:52:25.982919 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:52:25.982877 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:52:28.669428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:28.669393 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5"] Oct 31 13:52:29.124902 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:29.124862 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn"] Oct 31 13:52:29.140428 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:29.140393 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt"] Oct 31 13:52:30.313011 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.312969 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr"] Oct 31 13:52:30.313495 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.313279 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" podUID="b3774918-f6b4-42be-8f64-7b17743197fa" containerName="maas-api" containerID="cri-o://322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d" gracePeriod=30 Oct 31 13:52:30.591148 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.591118 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:52:30.686340 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.686298 5074 generic.go:356] "Generic (PLEG): container finished" podID="b3774918-f6b4-42be-8f64-7b17743197fa" containerID="322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d" exitCode=0 Oct 31 13:52:30.686536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.686343 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" Oct 31 13:52:30.686536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.686386 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" event={"ID":"b3774918-f6b4-42be-8f64-7b17743197fa","Type":"ContainerDied","Data":"322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d"} Oct 31 13:52:30.686536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.686442 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr" event={"ID":"b3774918-f6b4-42be-8f64-7b17743197fa","Type":"ContainerDied","Data":"f3ef7811a4818e056701405156f43e7afefb41b03bba802f9b8dcaaa17c93ea5"} Oct 31 13:52:30.686536 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.686469 5074 scope.go:122] "RemoveContainer" containerID="322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d" Oct 31 13:52:30.695834 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.695814 5074 scope.go:122] "RemoveContainer" containerID="322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d" Oct 31 13:52:30.696118 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:52:30.696096 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d\": container with ID starting with 322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d not found: ID does not exist" containerID="322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d" Oct 31 13:52:30.696203 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.696127 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d"} err="failed to get container status \"322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d\": rpc error: code = NotFound desc = could not find container \"322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d\": container with ID starting with 322098eca34887a9cce6888eb4e7669e90fdb1f3f15d6cf09fd9883ca266227d not found: ID does not exist" Oct 31 13:52:30.729384 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.729357 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-maas-api-tls\") pod \"b3774918-f6b4-42be-8f64-7b17743197fa\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " Oct 31 13:52:30.729531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.729403 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3774918-f6b4-42be-8f64-7b17743197fa-kube-api-access-7sqh9\" (UniqueName: \"kubernetes.io/projected/b3774918-f6b4-42be-8f64-7b17743197fa-kube-api-access-7sqh9\") pod \"b3774918-f6b4-42be-8f64-7b17743197fa\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " Oct 31 13:52:30.729531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.729425 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-metrics-certs\") pod \"b3774918-f6b4-42be-8f64-7b17743197fa\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " Oct 31 13:52:30.729531 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.729472 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3774918-f6b4-42be-8f64-7b17743197fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3774918-f6b4-42be-8f64-7b17743197fa-openshift-service-ca\") pod \"b3774918-f6b4-42be-8f64-7b17743197fa\" (UID: \"b3774918-f6b4-42be-8f64-7b17743197fa\") " Oct 31 13:52:30.729936 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.729914 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3774918-f6b4-42be-8f64-7b17743197fa-openshift-service-ca" pod "b3774918-f6b4-42be-8f64-7b17743197fa" (UID: "b3774918-f6b4-42be-8f64-7b17743197fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:30.730034 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.730023 5074 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3774918-f6b4-42be-8f64-7b17743197fa-openshift-service-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:30.731597 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.731573 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3774918-f6b4-42be-8f64-7b17743197fa-kube-api-access-7sqh9" pod "b3774918-f6b4-42be-8f64-7b17743197fa" (UID: "b3774918-f6b4-42be-8f64-7b17743197fa"). InnerVolumeSpecName "kube-api-access-7sqh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:30.732017 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.731996 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-maas-api-tls" pod "b3774918-f6b4-42be-8f64-7b17743197fa" (UID: "b3774918-f6b4-42be-8f64-7b17743197fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:52:30.732106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.732016 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-metrics-certs" pod "b3774918-f6b4-42be-8f64-7b17743197fa" (UID: "b3774918-f6b4-42be-8f64-7b17743197fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:52:30.830928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.830840 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-maas-api-tls\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:30.830928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.830875 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqh9\" (UniqueName: \"kubernetes.io/projected/b3774918-f6b4-42be-8f64-7b17743197fa-kube-api-access-7sqh9\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:30.830928 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:30.830887 5074 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3774918-f6b4-42be-8f64-7b17743197fa-metrics-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:31.009696 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:31.009666 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr"] Oct 31 13:52:31.015663 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:31.015623 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a5889-7c44759bd4-9qlrr"] Oct 31 13:52:31.936819 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:31.936782 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3774918-f6b4-42be-8f64-7b17743197fa" path="/var/lib/kubelet/pods/b3774918-f6b4-42be-8f64-7b17743197fa/volumes" Oct 31 13:52:32.178850 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:32.178813 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:52:34.125666 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:34.125588 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" podUID="13b95247-3a70-48dd-8984-d20c92b6744e" containerName="payload-processing" containerID="cri-o://2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65" gracePeriod=25 Oct 31 13:52:34.140884 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:34.140830 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" podUID="9586f702-cefc-4885-9a58-6f6a31cdde4f" containerName="payload-pre-processing" containerID="cri-o://f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558" gracePeriod=25 Oct 31 13:52:39.328052 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:39.328005 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" podUID="13b95247-3a70-48dd-8984-d20c92b6744e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:52:39.328474 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:39.328068 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" podUID="9586f702-cefc-4885-9a58-6f6a31cdde4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:52:43.689996 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:43.689936 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" podUID="5fa01446-4129-4cb7-9b37-75b2ab1a708a" containerName="main" containerID="cri-o://09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43" gracePeriod=45 Oct 31 13:52:43.953146 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:43.953087 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:52:44.067174 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067137 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-home\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-home\") pod \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " Oct 31 13:52:44.067342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067196 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-model-cache\") pod \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " Oct 31 13:52:44.067342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067246 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-dshm\") pod \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " Oct 31 13:52:44.067342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067299 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tls-certs\") pod \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " Oct 31 13:52:44.067342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067316 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tmp-dir\") pod \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " Oct 31 13:52:44.067580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067380 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa01446-4129-4cb7-9b37-75b2ab1a708a-kube-api-access-mpm6w\" (UniqueName: \"kubernetes.io/projected/5fa01446-4129-4cb7-9b37-75b2ab1a708a-kube-api-access-mpm6w\") pod \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\" (UID: \"5fa01446-4129-4cb7-9b37-75b2ab1a708a\") " Oct 31 13:52:44.067580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067439 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-home" pod "5fa01446-4129-4cb7-9b37-75b2ab1a708a" (UID: "5fa01446-4129-4cb7-9b37-75b2ab1a708a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:44.067580 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067477 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-model-cache" pod "5fa01446-4129-4cb7-9b37-75b2ab1a708a" (UID: "5fa01446-4129-4cb7-9b37-75b2ab1a708a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:44.067799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067708 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tmp-dir" pod "5fa01446-4129-4cb7-9b37-75b2ab1a708a" (UID: "5fa01446-4129-4cb7-9b37-75b2ab1a708a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:44.067799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067719 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:44.067799 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.067765 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:44.069502 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.069472 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-dshm" pod "5fa01446-4129-4cb7-9b37-75b2ab1a708a" (UID: "5fa01446-4129-4cb7-9b37-75b2ab1a708a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:44.069617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.069518 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tls-certs" pod "5fa01446-4129-4cb7-9b37-75b2ab1a708a" (UID: "5fa01446-4129-4cb7-9b37-75b2ab1a708a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:52:44.069617 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.069547 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa01446-4129-4cb7-9b37-75b2ab1a708a-kube-api-access-mpm6w" pod "5fa01446-4129-4cb7-9b37-75b2ab1a708a" (UID: "5fa01446-4129-4cb7-9b37-75b2ab1a708a"). InnerVolumeSpecName "kube-api-access-mpm6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:44.168284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.168247 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpm6w\" (UniqueName: \"kubernetes.io/projected/5fa01446-4129-4cb7-9b37-75b2ab1a708a-kube-api-access-mpm6w\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:44.168284 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.168283 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:44.168478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.168294 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:44.168478 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.168303 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa01446-4129-4cb7-9b37-75b2ab1a708a-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:44.762587 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.762548 5074 generic.go:356] "Generic (PLEG): container finished" podID="5fa01446-4129-4cb7-9b37-75b2ab1a708a" containerID="09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43" exitCode=0 Oct 31 13:52:44.763063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.762597 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" Oct 31 13:52:44.763063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.762647 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" event={"ID":"5fa01446-4129-4cb7-9b37-75b2ab1a708a","Type":"ContainerDied","Data":"09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43"} Oct 31 13:52:44.763063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.762694 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5" event={"ID":"5fa01446-4129-4cb7-9b37-75b2ab1a708a","Type":"ContainerDied","Data":"0b421580b29edcec0e19f83e33d34756d04a6c87198038ccba712434eb38eebf"} Oct 31 13:52:44.763063 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.762711 5074 scope.go:122] "RemoveContainer" containerID="09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43" Oct 31 13:52:44.775050 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.775031 5074 scope.go:122] "RemoveContainer" containerID="09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43" Oct 31 13:52:44.775325 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:52:44.775302 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43\": container with ID starting with 09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43 not found: ID does not exist" containerID="09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43" Oct 31 13:52:44.775379 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.775333 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43"} err="failed to get container status \"09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43\": rpc error: code = NotFound desc = could not find container \"09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43\": container with ID starting with 09922beaa3e1d88d2a21538f61e90fb0f6266553d4cf85dbee3091350c60ff43 not found: ID does not exist" Oct 31 13:52:44.788506 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.788483 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5"] Oct 31 13:52:44.792733 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:44.792707 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3a5889/simulator-single-main-3a5889-kserve-d498d7cdd-7pvz5"] Oct 31 13:52:45.942538 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:45.938400 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa01446-4129-4cb7-9b37-75b2ab1a708a" path="/var/lib/kubelet/pods/5fa01446-4129-4cb7-9b37-75b2ab1a708a/volumes" Oct 31 13:52:49.324999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:49.324958 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" podUID="9586f702-cefc-4885-9a58-6f6a31cdde4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:52:49.325412 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:49.324977 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" podUID="13b95247-3a70-48dd-8984-d20c92b6744e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:52:54.404445 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.404416 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:52:54.422711 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.422688 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:52:54.453029 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.452995 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9586f702-cefc-4885-9a58-6f6a31cdde4f-kube-api-access-tltpp\" (UniqueName: \"kubernetes.io/projected/9586f702-cefc-4885-9a58-6f6a31cdde4f-kube-api-access-tltpp\") pod \"9586f702-cefc-4885-9a58-6f6a31cdde4f\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " Oct 31 13:52:54.453180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.453058 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b95247-3a70-48dd-8984-d20c92b6744e-kube-api-access-hx5cx\" (UniqueName: \"kubernetes.io/projected/13b95247-3a70-48dd-8984-d20c92b6744e-kube-api-access-hx5cx\") pod \"13b95247-3a70-48dd-8984-d20c92b6744e\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " Oct 31 13:52:54.453180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.453132 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9586f702-cefc-4885-9a58-6f6a31cdde4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9586f702-cefc-4885-9a58-6f6a31cdde4f-plugins-config-volume\") pod \"9586f702-cefc-4885-9a58-6f6a31cdde4f\" (UID: \"9586f702-cefc-4885-9a58-6f6a31cdde4f\") " Oct 31 13:52:54.453180 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.453164 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b95247-3a70-48dd-8984-d20c92b6744e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b95247-3a70-48dd-8984-d20c92b6744e-plugins-config-volume\") pod \"13b95247-3a70-48dd-8984-d20c92b6744e\" (UID: \"13b95247-3a70-48dd-8984-d20c92b6744e\") " Oct 31 13:52:54.453779 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.453673 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9586f702-cefc-4885-9a58-6f6a31cdde4f-plugins-config-volume" pod "9586f702-cefc-4885-9a58-6f6a31cdde4f" (UID: "9586f702-cefc-4885-9a58-6f6a31cdde4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:54.453924 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.453890 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b95247-3a70-48dd-8984-d20c92b6744e-plugins-config-volume" pod "13b95247-3a70-48dd-8984-d20c92b6744e" (UID: "13b95247-3a70-48dd-8984-d20c92b6744e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:54.455253 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.455232 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9586f702-cefc-4885-9a58-6f6a31cdde4f-kube-api-access-tltpp" pod "9586f702-cefc-4885-9a58-6f6a31cdde4f" (UID: "9586f702-cefc-4885-9a58-6f6a31cdde4f"). InnerVolumeSpecName "kube-api-access-tltpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:54.455581 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.455558 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b95247-3a70-48dd-8984-d20c92b6744e-kube-api-access-hx5cx" pod "13b95247-3a70-48dd-8984-d20c92b6744e" (UID: "13b95247-3a70-48dd-8984-d20c92b6744e"). InnerVolumeSpecName "kube-api-access-hx5cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:54.554015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.553971 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tltpp\" (UniqueName: \"kubernetes.io/projected/9586f702-cefc-4885-9a58-6f6a31cdde4f-kube-api-access-tltpp\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:54.554015 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.554010 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5cx\" (UniqueName: \"kubernetes.io/projected/13b95247-3a70-48dd-8984-d20c92b6744e-kube-api-access-hx5cx\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:54.554230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.554025 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9586f702-cefc-4885-9a58-6f6a31cdde4f-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:54.554230 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.554039 5074 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b95247-3a70-48dd-8984-d20c92b6744e-plugins-config-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:54.821208 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.821168 5074 generic.go:356] "Generic (PLEG): container finished" podID="13b95247-3a70-48dd-8984-d20c92b6744e" containerID="2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65" exitCode=0 Oct 31 13:52:54.821424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.821214 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" Oct 31 13:52:54.821424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.821267 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" event={"ID":"13b95247-3a70-48dd-8984-d20c92b6744e","Type":"ContainerDied","Data":"2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65"} Oct 31 13:52:54.821424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.821320 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn" event={"ID":"13b95247-3a70-48dd-8984-d20c92b6744e","Type":"ContainerDied","Data":"1ddb6ff2cafa93b72d3ed3b10a4b50f871875b5fd1133cf5c042dcc33bc5f048"} Oct 31 13:52:54.821424 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.821345 5074 scope.go:122] "RemoveContainer" containerID="2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65" Oct 31 13:52:54.822890 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.822854 5074 generic.go:356] "Generic (PLEG): container finished" podID="9586f702-cefc-4885-9a58-6f6a31cdde4f" containerID="f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558" exitCode=0 Oct 31 13:52:54.823009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.822906 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" Oct 31 13:52:54.823009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.822937 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" event={"ID":"9586f702-cefc-4885-9a58-6f6a31cdde4f","Type":"ContainerDied","Data":"f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558"} Oct 31 13:52:54.823009 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.822971 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt" event={"ID":"9586f702-cefc-4885-9a58-6f6a31cdde4f","Type":"ContainerDied","Data":"c4783c6b004200f0c828221765d65573810df911c278fcb56d21c58758427a4e"} Oct 31 13:52:54.831955 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.831937 5074 scope.go:122] "RemoveContainer" containerID="2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65" Oct 31 13:52:54.832165 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:52:54.832145 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65\": container with ID starting with 2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65 not found: ID does not exist" containerID="2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65" Oct 31 13:52:54.832237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.832172 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65"} err="failed to get container status \"2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65\": rpc error: code = NotFound desc = could not find container \"2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65\": container with ID starting with 2f9d046d7cca1464f39c558ae2d2698f61cf1d607350c20db8d3bb6b607a3d65 not found: ID does not exist" Oct 31 13:52:54.832237 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.832187 5074 scope.go:122] "RemoveContainer" containerID="f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558" Oct 31 13:52:54.840494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.840477 5074 scope.go:122] "RemoveContainer" containerID="f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558" Oct 31 13:52:54.841282 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:52:54.841262 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558\": container with ID starting with f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558 not found: ID does not exist" containerID="f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558" Oct 31 13:52:54.841352 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.841292 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558"} err="failed to get container status \"f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558\": rpc error: code = NotFound desc = could not find container \"f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558\": container with ID starting with f2e3b37357a88f597355930b64c8e9148340e7a60dac35c947d4dfa799f26558 not found: ID does not exist" Oct 31 13:52:54.845089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.845057 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn"] Oct 31 13:52:54.849591 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.849567 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a5889-cc4947495gvzrn"] Oct 31 13:52:54.864370 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.864344 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt"] Oct 31 13:52:54.871682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:54.871658 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a5889-5bd4df7bdt"] Oct 31 13:52:55.936436 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:55.936404 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b95247-3a70-48dd-8984-d20c92b6744e" path="/var/lib/kubelet/pods/13b95247-3a70-48dd-8984-d20c92b6744e/volumes" Oct 31 13:52:55.936828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:52:55.936812 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9586f702-cefc-4885-9a58-6f6a31cdde4f" path="/var/lib/kubelet/pods/9586f702-cefc-4885-9a58-6f6a31cdde4f/volumes" Oct 31 13:52:55.981593 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:52:55.981558 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:53:04.368369 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:04.368328 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:53:07.533965 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:07.533913 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:53:09.911569 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.911529 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj"] Oct 31 13:53:09.911981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.911968 5074 cpu_manager.go:414] "containerMap: removing container" podUID="5fa01446-4129-4cb7-9b37-75b2ab1a708a" containerName="main" Oct 31 13:53:09.911981 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.911981 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa01446-4129-4cb7-9b37-75b2ab1a708a" containerName="main" Oct 31 13:53:09.912067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912000 5074 cpu_manager.go:414] "containerMap: removing container" podUID="9586f702-cefc-4885-9a58-6f6a31cdde4f" containerName="payload-pre-processing" Oct 31 13:53:09.912067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912006 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9586f702-cefc-4885-9a58-6f6a31cdde4f" containerName="payload-pre-processing" Oct 31 13:53:09.912067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912020 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b3774918-f6b4-42be-8f64-7b17743197fa" containerName="maas-api" Oct 31 13:53:09.912067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912025 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3774918-f6b4-42be-8f64-7b17743197fa" containerName="maas-api" Oct 31 13:53:09.912067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912045 5074 cpu_manager.go:414] "containerMap: removing container" podUID="13b95247-3a70-48dd-8984-d20c92b6744e" containerName="payload-processing" Oct 31 13:53:09.912067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912050 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b95247-3a70-48dd-8984-d20c92b6744e" containerName="payload-processing" Oct 31 13:53:09.912259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912110 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="9586f702-cefc-4885-9a58-6f6a31cdde4f" containerName="payload-pre-processing" Oct 31 13:53:09.912259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912121 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3774918-f6b4-42be-8f64-7b17743197fa" containerName="maas-api" Oct 31 13:53:09.912259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912130 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b95247-3a70-48dd-8984-d20c92b6744e" containerName="payload-processing" Oct 31 13:53:09.912259 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.912138 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa01446-4129-4cb7-9b37-75b2ab1a708a" containerName="main" Oct 31 13:53:09.915611 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.915591 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:09.923919 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.923148 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:53:09.924132 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.924108 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:53:09.924368 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.924320 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 13:53:09.924529 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.924514 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:53:09.926089 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:09.926068 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj"] Oct 31 13:53:10.009545 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.009488 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.009767 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.009710 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thgw\" (UniqueName: \"kubernetes.io/projected/1643c754-0dd7-4540-ab59-9d0f027133d0-kube-api-access-2thgw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.009839 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.009802 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.111487 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.111442 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.111723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.111553 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thgw\" (UniqueName: \"kubernetes.io/projected/1643c754-0dd7-4540-ab59-9d0f027133d0-kube-api-access-2thgw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.111723 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.111608 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.112328 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.112300 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.113878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.113856 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.124485 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.124459 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thgw\" (UniqueName: \"kubernetes.io/projected/1643c754-0dd7-4540-ab59-9d0f027133d0-kube-api-access-2thgw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.231041 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.231000 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:10.511312 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.511280 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj"] Oct 31 13:53:10.513108 ip-10-0-135-41 kubenswrapper[5074]: W1031 13:53:10.513080 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1643c754_0dd7_4540_ab59_9d0f027133d0.slice/crio-6152223d70e5e58440a8deb4f3ea85e7e02249c16967d3ee3444e5d7cef2ed4c WatchSource:0}: Error finding container 6152223d70e5e58440a8deb4f3ea85e7e02249c16967d3ee3444e5d7cef2ed4c: Status 404 returned error can't find the container with id 6152223d70e5e58440a8deb4f3ea85e7e02249c16967d3ee3444e5d7cef2ed4c Oct 31 13:53:10.905439 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.905402 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" event={"ID":"1643c754-0dd7-4540-ab59-9d0f027133d0","Type":"ContainerStarted","Data":"160e4bbc9ebc4ff479123a53d165debe1a282685b86927976f360d8d31a111ca"} Oct 31 13:53:10.905626 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.905461 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" event={"ID":"1643c754-0dd7-4540-ab59-9d0f027133d0","Type":"ContainerStarted","Data":"6152223d70e5e58440a8deb4f3ea85e7e02249c16967d3ee3444e5d7cef2ed4c"} Oct 31 13:53:10.924948 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:10.924892 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" podStartSLOduration=1.924878288 podStartE2EDuration="1.924878288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:10.923596328 +0000 UTC m=+4885.101314312" watchObservedRunningTime="2026-10-31 13:53:10.924878288 +0000 UTC m=+4885.102596272" Oct 31 13:53:11.911853 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:11.911815 5074 generic.go:356] "Generic (PLEG): container finished" podID="1643c754-0dd7-4540-ab59-9d0f027133d0" containerID="160e4bbc9ebc4ff479123a53d165debe1a282685b86927976f360d8d31a111ca" exitCode=0 Oct 31 13:53:11.912067 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:11.911891 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" event={"ID":"1643c754-0dd7-4540-ab59-9d0f027133d0","Type":"ContainerDied","Data":"160e4bbc9ebc4ff479123a53d165debe1a282685b86927976f360d8d31a111ca"} Oct 31 13:53:13.052082 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.052057 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:13.138704 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.138668 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1643c754-0dd7-4540-ab59-9d0f027133d0-kube-api-access-2thgw\" (UniqueName: \"kubernetes.io/projected/1643c754-0dd7-4540-ab59-9d0f027133d0-kube-api-access-2thgw\") pod \"1643c754-0dd7-4540-ab59-9d0f027133d0\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " Oct 31 13:53:13.138920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.138713 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-openshift-ca\") pod \"1643c754-0dd7-4540-ab59-9d0f027133d0\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " Oct 31 13:53:13.138920 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.138748 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-xks-ca\") pod \"1643c754-0dd7-4540-ab59-9d0f027133d0\" (UID: \"1643c754-0dd7-4540-ab59-9d0f027133d0\") " Oct 31 13:53:13.139105 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.139078 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-openshift-ca" pod "1643c754-0dd7-4540-ab59-9d0f027133d0" (UID: "1643c754-0dd7-4540-ab59-9d0f027133d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:13.140878 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.140856 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-xks-ca" pod "1643c754-0dd7-4540-ab59-9d0f027133d0" (UID: "1643c754-0dd7-4540-ab59-9d0f027133d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:13.140962 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.140934 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1643c754-0dd7-4540-ab59-9d0f027133d0-kube-api-access-2thgw" pod "1643c754-0dd7-4540-ab59-9d0f027133d0" (UID: "1643c754-0dd7-4540-ab59-9d0f027133d0"). InnerVolumeSpecName "kube-api-access-2thgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:13.240083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.240051 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:13.240083 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.240081 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1643c754-0dd7-4540-ab59-9d0f027133d0-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:13.240274 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.240091 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thgw\" (UniqueName: \"kubernetes.io/projected/1643c754-0dd7-4540-ab59-9d0f027133d0-kube-api-access-2thgw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:13.923147 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.923115 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" Oct 31 13:53:13.923336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.923153 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj" event={"ID":"1643c754-0dd7-4540-ab59-9d0f027133d0","Type":"ContainerDied","Data":"6152223d70e5e58440a8deb4f3ea85e7e02249c16967d3ee3444e5d7cef2ed4c"} Oct 31 13:53:13.923336 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:13.923199 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6152223d70e5e58440a8deb4f3ea85e7e02249c16967d3ee3444e5d7cef2ed4c" Oct 31 13:53:20.879673 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:20.879607 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj"] Oct 31 13:53:20.884923 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:20.884895 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3a5889-cqnbj"] Oct 31 13:53:21.255467 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:21.255419 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z"] Oct 31 13:53:21.255758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:21.255722 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" containerID="cri-o://6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a" gracePeriod=30 Oct 31 13:53:21.937196 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:21.937157 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1643c754-0dd7-4540-ab59-9d0f027133d0" path="/var/lib/kubelet/pods/1643c754-0dd7-4540-ab59-9d0f027133d0/volumes" Oct 31 13:53:22.174070 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:22.174028 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:53:25.983110 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:53:25.983068 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:53:26.525181 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.525117 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:53:26.579366 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.579333 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:53:26.677527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677481 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-data\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.677527 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677533 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-kube-api-access-cts2m\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-kube-api-access-cts2m\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.677806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677580 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-istio-token\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-istio-token\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.677806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677607 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf502e76-d039-40bc-943b-99d3778bd071-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e76-d039-40bc-943b-99d3778bd071-istiod-ca-cert\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.677806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677645 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-envoy\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.677806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677689 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-socket\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.677806 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677721 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-certs\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.678069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677824 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf502e76-d039-40bc-943b-99d3778bd071-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf502e76-d039-40bc-943b-99d3778bd071-istio-podinfo\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.678069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677875 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-data" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:26.678069 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.677897 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-credential-socket\") pod \"bf502e76-d039-40bc-943b-99d3778bd071\" (UID: \"bf502e76-d039-40bc-943b-99d3778bd071\") " Oct 31 13:53:26.678229 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.678136 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-credential-socket" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:26.678286 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.678249 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf502e76-d039-40bc-943b-99d3778bd071-istiod-ca-cert" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:26.678416 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.678394 5074 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-data\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.678492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.678424 5074 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e76-d039-40bc-943b-99d3778bd071-istiod-ca-cert\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.678492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.678429 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-certs" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:26.678492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.678440 5074 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-credential-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.678492 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.678132 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-socket" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:26.680463 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.680431 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-envoy" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:26.680608 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.680578 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf502e76-d039-40bc-943b-99d3778bd071-istio-podinfo" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:53:26.680729 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.680681 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-kube-api-access-cts2m" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "kube-api-access-cts2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:26.680783 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.680765 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-istio-token" pod "bf502e76-d039-40bc-943b-99d3778bd071" (UID: "bf502e76-d039-40bc-943b-99d3778bd071"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:26.779684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.779559 5074 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf502e76-d039-40bc-943b-99d3778bd071-istio-podinfo\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.779684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.779606 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cts2m\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-kube-api-access-cts2m\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.779684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.779622 5074 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf502e76-d039-40bc-943b-99d3778bd071-istio-token\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.779684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.779661 5074 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-istio-envoy\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.779684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.779671 5074 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-socket\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:26.779684 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:26.779680 5074 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf502e76-d039-40bc-943b-99d3778bd071-workload-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:27.007411 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.007376 5074 generic.go:356] "Generic (PLEG): container finished" podID="bf502e76-d039-40bc-943b-99d3778bd071" containerID="6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a" exitCode=0 Oct 31 13:53:27.007954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.007435 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" Oct 31 13:53:27.007954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.007464 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" event={"ID":"bf502e76-d039-40bc-943b-99d3778bd071","Type":"ContainerDied","Data":"6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a"} Oct 31 13:53:27.007954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.007537 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z" event={"ID":"bf502e76-d039-40bc-943b-99d3778bd071","Type":"ContainerDied","Data":"9ba8d3fba428ac586713c3086f45147b54f59a81df9d32bef43386a7d8bd06c3"} Oct 31 13:53:27.007954 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.007569 5074 scope.go:122] "RemoveContainer" containerID="6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a" Oct 31 13:53:27.019151 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.019133 5074 scope.go:122] "RemoveContainer" containerID="43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d" Oct 31 13:53:27.052031 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.051985 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z"] Oct 31 13:53:27.057915 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.057851 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3a5889-openshift-default-5c8b869645zpw5z"] Oct 31 13:53:27.062828 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.062804 5074 scope.go:122] "RemoveContainer" containerID="6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a" Oct 31 13:53:27.063245 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:53:27.063217 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a\": container with ID starting with 6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a not found: ID does not exist" containerID="6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a" Oct 31 13:53:27.063329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.063257 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a"} err="failed to get container status \"6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a\": rpc error: code = NotFound desc = could not find container \"6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a\": container with ID starting with 6f0b998757760422b8b92ec5a376353e17856605a85246c85924b37983e9619a not found: ID does not exist" Oct 31 13:53:27.063329 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.063281 5074 scope.go:122] "RemoveContainer" containerID="43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d" Oct 31 13:53:27.063600 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:53:27.063570 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d\": container with ID starting with 43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d not found: ID does not exist" containerID="43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d" Oct 31 13:53:27.063682 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.063596 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d"} err="failed to get container status \"43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d\": rpc error: code = NotFound desc = could not find container \"43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d\": container with ID starting with 43031c15ff6dbcd9c3322dd90d5305b8508e676e2e6ab21e83f2c19227c5df6d not found: ID does not exist" Oct 31 13:53:27.941001 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:27.940959 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf502e76-d039-40bc-943b-99d3778bd071" path="/var/lib/kubelet/pods/bf502e76-d039-40bc-943b-99d3778bd071/volumes" Oct 31 13:53:45.549053 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:45.549008 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:53:48.730788 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:48.730741 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:53:48.731409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:48.731372 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:53:55.982295 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:53:55.982256 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:53:57.182755 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:57.182721 5074 scope.go:122] "RemoveContainer" containerID="8165b5a3ed1ba3b29dbc9affb0090418ab6ba0919554c465aafb4025494f6641" Oct 31 13:53:59.765876 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:53:59.765838 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:54:11.463803 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:54:11.463759 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:54:16.461313 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:54:16.461270 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:54:25.982494 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:54:25.982449 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:54:26.266409 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:54:26.266327 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:54:30.750662 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:54:30.745838 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:54:43.760231 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:54:43.760190 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:54:55.982285 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:54:55.982247 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:55:25.981493 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:55:25.981457 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:55:30.948156 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:55:30.948118 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:55:33.730829 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:55:33.730776 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:55:33.731521 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:55:33.731472 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:55.981773 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:55:55.981730 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:55:58.022571 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:55:58.022529 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk"] Oct 31 13:55:58.024999 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:55:58.024973 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd1d38c-fn5rk"] Oct 31 13:55:59.936191 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:55:59.936155 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb73cf2-54df-4a15-afe7-33b14cf5403d" path="/var/lib/kubelet/pods/6cb73cf2-54df-4a15-afe7-33b14cf5403d/volumes" Oct 31 13:56:14.021925 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:56:14.021890 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p"] Oct 31 13:56:14.024758 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:56:14.024732 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d15814f-xz78p"] Oct 31 13:56:15.937559 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:56:15.937521 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0ae20c-bd2b-4f78-95f4-8be35fe619c7" path="/var/lib/kubelet/pods/7f0ae20c-bd2b-4f78-95f4-8be35fe619c7/volumes" Oct 31 13:56:25.982090 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:56:25.982034 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:56:47.299667 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:56:47.299615 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:56:47.334585 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:56:47.334557 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 13:56:55.981416 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:56:55.981375 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:56:57.220106 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:56:57.220069 5074 scope.go:122] "RemoveContainer" containerID="01241694894cf3b51cc9f010648ca842db87d991d101cf0a65c3d81ea8c7b276" Oct 31 13:57:08.033494 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:08.033450 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:57:12.408781 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:12.408733 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:57:12.409554 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:12.409515 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:57:12.991491 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:12.991453 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:57:25.982417 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:57:25.982378 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:57:28.354141 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:28.354099 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:57:32.896886 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:32.896843 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:57:44.571885 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:44.571834 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:57:48.736657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:48.732748 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-szf5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:57:48.736657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:48.732834 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-szf5x" podUID="89558c79-c272-4647-a36f-8130cb513976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:57:55.982395 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:57:55.982356 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:57:57.240836 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:57.240802 5074 scope.go:122] "RemoveContainer" containerID="10e7baabea4aad54dcf342eef7333bb09ef35b3983f6cebe54b266cfa5a906fd" Oct 31 13:57:57.444292 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:57:57.444253 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:58:21.784749 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:58:21.784709 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:58:25.981802 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:58:25.981768 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:58:32.134843 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:58:32.134804 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:58:35.979193 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:58:35.979149 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:58:51.979342 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:58:51.979303 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:58:55.751865 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:58:55.751825 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:58:56.056141 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:58:56.055985 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:58:59.180244 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:58:59.180196 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:59:10.039160 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:10.039116 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:59:12.411657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:12.410747 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:59:12.411657 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:12.410836 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:59:23.780656 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:23.780600 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:59:25.987714 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:59:25.987662 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:59:27.527346 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:27.527289 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:59:30.877642 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:30.877593 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:59:49.069314 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:49.069268 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:59:53.405888 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:53.405843 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 13:59:55.982542 ip-10-0-135-41 kubenswrapper[5074]: E1031 13:59:55.982504 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 13:59:57.270480 ip-10-0-135-41 kubenswrapper[5074]: I1031 13:59:57.270442 5074 scope.go:122] "RemoveContainer" containerID="160e4bbc9ebc4ff479123a53d165debe1a282685b86927976f360d8d31a111ca" Oct 31 14:00:06.777376 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:00:06.777336 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:00:10.541804 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:00:10.541767 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:00:13.739544 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:00:13.739505 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:00:25.982520 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:00:25.982480 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:00:28.559487 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:00:28.559448 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:00:38.750705 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:00:38.750661 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:00:54.542819 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:00:54.542777 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:00:56.027314 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:00:56.027087 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:00:58.533280 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:00:58.533231 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:01:06.794266 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:06.794223 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:01:20.664180 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664136 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77"] Oct 31 14:01:20.664792 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664770 5074 cpu_manager.go:414] "containerMap: removing container" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" Oct 31 14:01:20.664853 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664796 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" Oct 31 14:01:20.664896 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664854 5074 cpu_manager.go:414] "containerMap: removing container" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" Oct 31 14:01:20.664896 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664862 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" Oct 31 14:01:20.664896 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664892 5074 cpu_manager.go:414] "containerMap: removing container" podUID="1643c754-0dd7-4540-ab59-9d0f027133d0" containerName="revoke-keys" Oct 31 14:01:20.664996 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664901 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1643c754-0dd7-4540-ab59-9d0f027133d0" containerName="revoke-keys" Oct 31 14:01:20.665030 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.664994 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="1643c754-0dd7-4540-ab59-9d0f027133d0" containerName="revoke-keys" Oct 31 14:01:20.665030 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.665010 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" Oct 31 14:01:20.668893 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.668866 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.674389 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.674341 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5d3f86\"/\"openshift-service-ca.crt\"" Oct 31 14:01:20.674854 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.674825 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77"] Oct 31 14:01:20.676426 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.676378 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5d3f86\"/\"kube-root-ca.crt\"" Oct 31 14:01:20.709153 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.709115 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5d3f86\"/\"default-dockercfg-zx56d\"" Oct 31 14:01:20.709153 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.709122 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5d3f86\"/\"simulator-single-main-5d3f86-kserve-self-signed-certs\"" Oct 31 14:01:20.781139 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.781086 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-dshm\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.781344 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.781241 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-home\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.781344 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.781319 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-model-cache\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.781463 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.781343 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-tmp-dir\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.781463 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.781368 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1098c5-293c-4721-9e09-0d84cbe17e33-tls-certs\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.781594 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.781539 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdj82\" (UniqueName: \"kubernetes.io/projected/fe1098c5-293c-4721-9e09-0d84cbe17e33-kube-api-access-rdj82\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.882548 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.882498 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-home\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.882795 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.882587 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-model-cache\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.882795 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.882621 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-tmp-dir\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.882795 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.882671 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1098c5-293c-4721-9e09-0d84cbe17e33-tls-certs\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.882795 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.882728 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdj82\" (UniqueName: \"kubernetes.io/projected/fe1098c5-293c-4721-9e09-0d84cbe17e33-kube-api-access-rdj82\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.882795 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.882784 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-dshm\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.883105 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.883039 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-home\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.883105 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.883090 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-model-cache\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.883261 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.883239 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-tmp-dir\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.885347 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.885323 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1098c5-293c-4721-9e09-0d84cbe17e33-tls-certs\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.885457 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.885344 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-dshm\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:20.897925 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:20.897900 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdj82\" (UniqueName: \"kubernetes.io/projected/fe1098c5-293c-4721-9e09-0d84cbe17e33-kube-api-access-rdj82\") pod \"simulator-single-main-5d3f86-kserve-54799d8d48-jqv77\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:21.013381 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:21.013342 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:21.299192 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:21.299157 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77"] Oct 31 14:01:21.301275 ip-10-0-135-41 kubenswrapper[5074]: W1031 14:01:21.301241 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe1098c5_293c_4721_9e09_0d84cbe17e33.slice/crio-d527a4d0b79850691fd662b22b8181c99822f9251eb6a9071bebedf43867d9dd WatchSource:0}: Error finding container d527a4d0b79850691fd662b22b8181c99822f9251eb6a9071bebedf43867d9dd: Status 404 returned error can't find the container with id d527a4d0b79850691fd662b22b8181c99822f9251eb6a9071bebedf43867d9dd Oct 31 14:01:21.701239 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:21.701135 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" event={"ID":"fe1098c5-293c-4721-9e09-0d84cbe17e33","Type":"ContainerStarted","Data":"81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764"} Oct 31 14:01:21.701239 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:21.701182 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" event={"ID":"fe1098c5-293c-4721-9e09-0d84cbe17e33","Type":"ContainerStarted","Data":"d527a4d0b79850691fd662b22b8181c99822f9251eb6a9071bebedf43867d9dd"} Oct 31 14:01:21.721264 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:21.721204 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" podStartSLOduration=1.721191745 podStartE2EDuration="1.721191745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:21.720285173 +0000 UTC m=+5375.898003157" watchObservedRunningTime="2026-10-31 14:01:21.721191745 +0000 UTC m=+5375.898909729" Oct 31 14:01:22.013978 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:22.013925 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:22.479499 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:22.479453 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:01:25.982199 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:01:25.982159 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:01:31.014312 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:31.014271 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:31.027414 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:31.027375 5074 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:31.765468 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:31.765436 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:01:38.760940 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:38.760899 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:01:47.329854 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:47.329741 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 14:01:47.364678 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:47.364616 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 14:01:50.128444 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.128380 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n"] Oct 31 14:01:50.129092 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.129068 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf502e76-d039-40bc-943b-99d3778bd071" containerName="istio-proxy" Oct 31 14:01:50.131866 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.131844 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.140689 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.140665 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:01:50.140808 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.140792 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:01:50.141011 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.140997 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:01:50.141104 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.141046 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:01:50.141666 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.141620 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n"] Oct 31 14:01:50.183830 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.183788 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.183830 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.183829 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pknr\" (UniqueName: \"kubernetes.io/projected/b567bef2-e9db-42de-82a4-e2dea9f7a636-kube-api-access-9pknr\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.184030 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.183963 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.285476 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.285434 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.285710 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.285481 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pknr\" (UniqueName: \"kubernetes.io/projected/b567bef2-e9db-42de-82a4-e2dea9f7a636-kube-api-access-9pknr\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.285710 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.285552 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.286286 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.286260 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.287845 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.287823 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.298441 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.298413 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pknr\" (UniqueName: \"kubernetes.io/projected/b567bef2-e9db-42de-82a4-e2dea9f7a636-kube-api-access-9pknr\") pod \"maas-api-revoke-sub-771298d6-lvk5n\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.446348 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.446262 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:50.721477 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.721452 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n"] Oct 31 14:01:50.723594 ip-10-0-135-41 kubenswrapper[5074]: W1031 14:01:50.723566 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb567bef2_e9db_42de_82a4_e2dea9f7a636.slice/crio-867354c5eaec42c8cc1145a7e9c33bfb7d8d64660357804c9d4e7d92c928a578 WatchSource:0}: Error finding container 867354c5eaec42c8cc1145a7e9c33bfb7d8d64660357804c9d4e7d92c928a578: Status 404 returned error can't find the container with id 867354c5eaec42c8cc1145a7e9c33bfb7d8d64660357804c9d4e7d92c928a578 Oct 31 14:01:50.853013 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:50.852971 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" event={"ID":"b567bef2-e9db-42de-82a4-e2dea9f7a636","Type":"ContainerStarted","Data":"867354c5eaec42c8cc1145a7e9c33bfb7d8d64660357804c9d4e7d92c928a578"} Oct 31 14:01:51.859877 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:51.859787 5074 generic.go:356] "Generic (PLEG): container finished" podID="b567bef2-e9db-42de-82a4-e2dea9f7a636" containerID="48fd60e43aa099b4cc84ee111bc7a797d8c788a1df22bf1dd0f9212d5cd34ee3" exitCode=0 Oct 31 14:01:51.860271 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:51.859884 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" event={"ID":"b567bef2-e9db-42de-82a4-e2dea9f7a636","Type":"ContainerDied","Data":"48fd60e43aa099b4cc84ee111bc7a797d8c788a1df22bf1dd0f9212d5cd34ee3"} Oct 31 14:01:53.000745 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.000711 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:53.114542 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.114506 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-xks-ca\") pod \"b567bef2-e9db-42de-82a4-e2dea9f7a636\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " Oct 31 14:01:53.114742 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.114610 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b567bef2-e9db-42de-82a4-e2dea9f7a636-kube-api-access-9pknr\" (UniqueName: \"kubernetes.io/projected/b567bef2-e9db-42de-82a4-e2dea9f7a636-kube-api-access-9pknr\") pod \"b567bef2-e9db-42de-82a4-e2dea9f7a636\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " Oct 31 14:01:53.114742 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.114727 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-openshift-ca\") pod \"b567bef2-e9db-42de-82a4-e2dea9f7a636\" (UID: \"b567bef2-e9db-42de-82a4-e2dea9f7a636\") " Oct 31 14:01:53.115104 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.115079 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-openshift-ca" pod "b567bef2-e9db-42de-82a4-e2dea9f7a636" (UID: "b567bef2-e9db-42de-82a4-e2dea9f7a636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:53.116721 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.116669 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-xks-ca" pod "b567bef2-e9db-42de-82a4-e2dea9f7a636" (UID: "b567bef2-e9db-42de-82a4-e2dea9f7a636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:53.116721 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.116696 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b567bef2-e9db-42de-82a4-e2dea9f7a636-kube-api-access-9pknr" pod "b567bef2-e9db-42de-82a4-e2dea9f7a636" (UID: "b567bef2-e9db-42de-82a4-e2dea9f7a636"). InnerVolumeSpecName "kube-api-access-9pknr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:53.216359 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.216325 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pknr\" (UniqueName: \"kubernetes.io/projected/b567bef2-e9db-42de-82a4-e2dea9f7a636-kube-api-access-9pknr\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:53.216359 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.216357 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:53.216566 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.216369 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b567bef2-e9db-42de-82a4-e2dea9f7a636-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:53.871498 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.871467 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" Oct 31 14:01:53.871703 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.871463 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n" event={"ID":"b567bef2-e9db-42de-82a4-e2dea9f7a636","Type":"ContainerDied","Data":"867354c5eaec42c8cc1145a7e9c33bfb7d8d64660357804c9d4e7d92c928a578"} Oct 31 14:01:53.871703 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:01:53.871602 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="867354c5eaec42c8cc1145a7e9c33bfb7d8d64660357804c9d4e7d92c928a578" Oct 31 14:01:55.982603 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:01:55.982555 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:02:01.774585 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:01.774548 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:02:06.060973 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:06.060934 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77"] Oct 31 14:02:09.949683 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:09.949645 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:02:21.081043 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.081001 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" podUID="fe1098c5-293c-4721-9e09-0d84cbe17e33" containerName="main" containerID="cri-o://81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764" gracePeriod=45 Oct 31 14:02:21.329758 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.329732 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:02:21.391216 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391142 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-dshm\") pod \"fe1098c5-293c-4721-9e09-0d84cbe17e33\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " Oct 31 14:02:21.391384 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391221 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-home\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-home\") pod \"fe1098c5-293c-4721-9e09-0d84cbe17e33\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " Oct 31 14:02:21.391384 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391268 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1098c5-293c-4721-9e09-0d84cbe17e33-kube-api-access-rdj82\" (UniqueName: \"kubernetes.io/projected/fe1098c5-293c-4721-9e09-0d84cbe17e33-kube-api-access-rdj82\") pod \"fe1098c5-293c-4721-9e09-0d84cbe17e33\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " Oct 31 14:02:21.391384 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391297 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1098c5-293c-4721-9e09-0d84cbe17e33-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1098c5-293c-4721-9e09-0d84cbe17e33-tls-certs\") pod \"fe1098c5-293c-4721-9e09-0d84cbe17e33\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " Oct 31 14:02:21.391575 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391392 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-model-cache\") pod \"fe1098c5-293c-4721-9e09-0d84cbe17e33\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " Oct 31 14:02:21.391575 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391457 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-tmp-dir\") pod \"fe1098c5-293c-4721-9e09-0d84cbe17e33\" (UID: \"fe1098c5-293c-4721-9e09-0d84cbe17e33\") " Oct 31 14:02:21.391575 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391535 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-home" pod "fe1098c5-293c-4721-9e09-0d84cbe17e33" (UID: "fe1098c5-293c-4721-9e09-0d84cbe17e33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:21.391823 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391787 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-model-cache" pod "fe1098c5-293c-4721-9e09-0d84cbe17e33" (UID: "fe1098c5-293c-4721-9e09-0d84cbe17e33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:21.391982 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391900 5074 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-home\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:21.391982 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391922 5074 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-model-cache\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:21.391982 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.391929 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-tmp-dir" pod "fe1098c5-293c-4721-9e09-0d84cbe17e33" (UID: "fe1098c5-293c-4721-9e09-0d84cbe17e33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:21.393613 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.393589 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1098c5-293c-4721-9e09-0d84cbe17e33-tls-certs" pod "fe1098c5-293c-4721-9e09-0d84cbe17e33" (UID: "fe1098c5-293c-4721-9e09-0d84cbe17e33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:21.393880 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.393846 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-dshm" pod "fe1098c5-293c-4721-9e09-0d84cbe17e33" (UID: "fe1098c5-293c-4721-9e09-0d84cbe17e33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:21.393939 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.393885 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1098c5-293c-4721-9e09-0d84cbe17e33-kube-api-access-rdj82" pod "fe1098c5-293c-4721-9e09-0d84cbe17e33" (UID: "fe1098c5-293c-4721-9e09-0d84cbe17e33"). InnerVolumeSpecName "kube-api-access-rdj82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:21.493263 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.493227 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdj82\" (UniqueName: \"kubernetes.io/projected/fe1098c5-293c-4721-9e09-0d84cbe17e33-kube-api-access-rdj82\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:21.493263 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.493257 5074 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1098c5-293c-4721-9e09-0d84cbe17e33-tls-certs\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:21.493263 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.493266 5074 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-tmp-dir\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:21.493503 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:21.493276 5074 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1098c5-293c-4721-9e09-0d84cbe17e33-dshm\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:22.013325 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.013291 5074 generic.go:356] "Generic (PLEG): container finished" podID="fe1098c5-293c-4721-9e09-0d84cbe17e33" containerID="81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764" exitCode=0 Oct 31 14:02:22.013537 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.013337 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" Oct 31 14:02:22.013537 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.013369 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" event={"ID":"fe1098c5-293c-4721-9e09-0d84cbe17e33","Type":"ContainerDied","Data":"81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764"} Oct 31 14:02:22.013537 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.013418 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77" event={"ID":"fe1098c5-293c-4721-9e09-0d84cbe17e33","Type":"ContainerDied","Data":"d527a4d0b79850691fd662b22b8181c99822f9251eb6a9071bebedf43867d9dd"} Oct 31 14:02:22.013537 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.013436 5074 scope.go:122] "RemoveContainer" containerID="81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764" Oct 31 14:02:22.022546 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.022523 5074 scope.go:122] "RemoveContainer" containerID="81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764" Oct 31 14:02:22.022844 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:02:22.022822 5074 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764\": container with ID starting with 81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764 not found: ID does not exist" containerID="81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764" Oct 31 14:02:22.022925 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.022856 5074 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764"} err="failed to get container status \"81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764\": rpc error: code = NotFound desc = could not find container \"81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764\": container with ID starting with 81d2b05196ac09e47537182b1b115800b054a5875cfd05e3a7d14d39462d3764 not found: ID does not exist" Oct 31 14:02:22.031162 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.031138 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77"] Oct 31 14:02:22.035510 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:22.035489 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-5d3f86/simulator-single-main-5d3f86-kserve-54799d8d48-jqv77"] Oct 31 14:02:23.936986 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:23.936941 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1098c5-293c-4721-9e09-0d84cbe17e33" path="/var/lib/kubelet/pods/fe1098c5-293c-4721-9e09-0d84cbe17e33/volumes" Oct 31 14:02:25.981801 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:02:25.981752 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:02:42.362277 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:42.362231 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:02:47.134460 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:47.134423 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:02:48.232928 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.232895 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl"] Oct 31 14:02:48.233365 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.233344 5074 cpu_manager.go:414] "containerMap: removing container" podUID="fe1098c5-293c-4721-9e09-0d84cbe17e33" containerName="main" Oct 31 14:02:48.233365 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.233357 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1098c5-293c-4721-9e09-0d84cbe17e33" containerName="main" Oct 31 14:02:48.233454 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.233369 5074 cpu_manager.go:414] "containerMap: removing container" podUID="b567bef2-e9db-42de-82a4-e2dea9f7a636" containerName="revoke-keys" Oct 31 14:02:48.233454 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.233375 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b567bef2-e9db-42de-82a4-e2dea9f7a636" containerName="revoke-keys" Oct 31 14:02:48.233454 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.233443 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1098c5-293c-4721-9e09-0d84cbe17e33" containerName="main" Oct 31 14:02:48.233454 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.233452 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="b567bef2-e9db-42de-82a4-e2dea9f7a636" containerName="revoke-keys" Oct 31 14:02:48.236662 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.236643 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.240508 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.240482 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:02:48.243340 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.243319 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:02:48.245248 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.245222 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cqrz\"" Oct 31 14:02:48.245495 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.245473 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:02:48.247734 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.247712 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl"] Oct 31 14:02:48.342168 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.342129 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjvv\" (UniqueName: \"kubernetes.io/projected/29010444-d00c-4f5e-91d9-706a5f8900db-kube-api-access-8rjvv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.342365 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.342211 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.342365 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.342312 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.443876 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.443836 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjvv\" (UniqueName: \"kubernetes.io/projected/29010444-d00c-4f5e-91d9-706a5f8900db-kube-api-access-8rjvv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.444074 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.443910 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.444074 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.443932 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.444583 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.444555 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.446354 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.446324 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.456092 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.456068 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjvv\" (UniqueName: \"kubernetes.io/projected/29010444-d00c-4f5e-91d9-706a5f8900db-kube-api-access-8rjvv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.552568 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.552485 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:48.830453 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:48.830321 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl"] Oct 31 14:02:48.832834 ip-10-0-135-41 kubenswrapper[5074]: W1031 14:02:48.832807 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29010444_d00c_4f5e_91d9_706a5f8900db.slice/crio-3f3a715696c5d14ed712b08d66a65cf24982bbb79e39e7a5be1ac65343010b30 WatchSource:0}: Error finding container 3f3a715696c5d14ed712b08d66a65cf24982bbb79e39e7a5be1ac65343010b30: Status 404 returned error can't find the container with id 3f3a715696c5d14ed712b08d66a65cf24982bbb79e39e7a5be1ac65343010b30 Oct 31 14:02:49.151703 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:49.151595 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" event={"ID":"29010444-d00c-4f5e-91d9-706a5f8900db","Type":"ContainerStarted","Data":"73a5d135fb1a84fb803eee30c07ca5979da8ceae04df1b9d80cddd316eb158bd"} Oct 31 14:02:49.151703 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:49.151687 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" event={"ID":"29010444-d00c-4f5e-91d9-706a5f8900db","Type":"ContainerStarted","Data":"3f3a715696c5d14ed712b08d66a65cf24982bbb79e39e7a5be1ac65343010b30"} Oct 31 14:02:49.170806 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:49.170755 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" podStartSLOduration=1.170737671 podStartE2EDuration="1.170737671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:49.169097375 +0000 UTC m=+5463.346815358" watchObservedRunningTime="2026-10-31 14:02:49.170737671 +0000 UTC m=+5463.348455659" Oct 31 14:02:50.158067 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:50.158008 5074 generic.go:356] "Generic (PLEG): container finished" podID="29010444-d00c-4f5e-91d9-706a5f8900db" containerID="73a5d135fb1a84fb803eee30c07ca5979da8ceae04df1b9d80cddd316eb158bd" exitCode=0 Oct 31 14:02:50.158475 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:50.158092 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" event={"ID":"29010444-d00c-4f5e-91d9-706a5f8900db","Type":"ContainerDied","Data":"73a5d135fb1a84fb803eee30c07ca5979da8ceae04df1b9d80cddd316eb158bd"} Oct 31 14:02:51.304455 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.304426 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:51.372553 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.372517 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-xks-ca\") pod \"29010444-d00c-4f5e-91d9-706a5f8900db\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " Oct 31 14:02:51.372743 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.372680 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29010444-d00c-4f5e-91d9-706a5f8900db-kube-api-access-8rjvv\" (UniqueName: \"kubernetes.io/projected/29010444-d00c-4f5e-91d9-706a5f8900db-kube-api-access-8rjvv\") pod \"29010444-d00c-4f5e-91d9-706a5f8900db\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " Oct 31 14:02:51.372743 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.372715 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-openshift-ca\") pod \"29010444-d00c-4f5e-91d9-706a5f8900db\" (UID: \"29010444-d00c-4f5e-91d9-706a5f8900db\") " Oct 31 14:02:51.373124 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.373099 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-openshift-ca" pod "29010444-d00c-4f5e-91d9-706a5f8900db" (UID: "29010444-d00c-4f5e-91d9-706a5f8900db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:51.374809 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.374784 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-xks-ca" pod "29010444-d00c-4f5e-91d9-706a5f8900db" (UID: "29010444-d00c-4f5e-91d9-706a5f8900db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:51.374915 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.374788 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29010444-d00c-4f5e-91d9-706a5f8900db-kube-api-access-8rjvv" pod "29010444-d00c-4f5e-91d9-706a5f8900db" (UID: "29010444-d00c-4f5e-91d9-706a5f8900db"). InnerVolumeSpecName "kube-api-access-8rjvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:51.474521 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.474485 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-openshift-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.474521 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.474517 5074 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29010444-d00c-4f5e-91d9-706a5f8900db-maas-api-xks-ca\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.474740 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:51.474528 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjvv\" (UniqueName: \"kubernetes.io/projected/29010444-d00c-4f5e-91d9-706a5f8900db-kube-api-access-8rjvv\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:52.169142 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:52.169109 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" Oct 31 14:02:52.169379 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:52.169145 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl" event={"ID":"29010444-d00c-4f5e-91d9-706a5f8900db","Type":"ContainerDied","Data":"3f3a715696c5d14ed712b08d66a65cf24982bbb79e39e7a5be1ac65343010b30"} Oct 31 14:02:52.169379 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:52.169194 5074 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3a715696c5d14ed712b08d66a65cf24982bbb79e39e7a5be1ac65343010b30" Oct 31 14:02:55.981943 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:02:55.981906 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:02:59.359785 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:59.359749 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl"] Oct 31 14:02:59.364388 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:59.364363 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d3f86-zwpnl"] Oct 31 14:02:59.938012 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:02:59.937961 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29010444-d00c-4f5e-91d9-706a5f8900db" path="/var/lib/kubelet/pods/29010444-d00c-4f5e-91d9-706a5f8900db/volumes" Oct 31 14:03:00.748567 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:00.748522 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:03:11.956970 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:11.956930 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:03:15.958256 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:15.958217 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:03:16.673498 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:16.673461 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j"] Oct 31 14:03:16.673714 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:16.673672 5074 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" podUID="87dfb15b-fa89-461c-b36a-bceb692300c0" containerName="manager" containerID="cri-o://91369d75f398460a3e9e1b0e44645d6884f03af6bafec1b8d5672a80bb693649" gracePeriod=10 Oct 31 14:03:17.317090 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.317053 5074 generic.go:356] "Generic (PLEG): container finished" podID="87dfb15b-fa89-461c-b36a-bceb692300c0" containerID="91369d75f398460a3e9e1b0e44645d6884f03af6bafec1b8d5672a80bb693649" exitCode=0 Oct 31 14:03:17.317466 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.317126 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" event={"ID":"87dfb15b-fa89-461c-b36a-bceb692300c0","Type":"ContainerDied","Data":"91369d75f398460a3e9e1b0e44645d6884f03af6bafec1b8d5672a80bb693649"} Oct 31 14:03:17.440309 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.440279 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 14:03:17.476857 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.476820 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87dfb15b-fa89-461c-b36a-bceb692300c0-kube-api-access-d2bcw\" (UniqueName: \"kubernetes.io/projected/87dfb15b-fa89-461c-b36a-bceb692300c0-kube-api-access-d2bcw\") pod \"87dfb15b-fa89-461c-b36a-bceb692300c0\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " Oct 31 14:03:17.477063 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.476891 5074 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87dfb15b-fa89-461c-b36a-bceb692300c0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dfb15b-fa89-461c-b36a-bceb692300c0-extensions-socket-volume\") pod \"87dfb15b-fa89-461c-b36a-bceb692300c0\" (UID: \"87dfb15b-fa89-461c-b36a-bceb692300c0\") " Oct 31 14:03:17.477623 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.477591 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87dfb15b-fa89-461c-b36a-bceb692300c0-extensions-socket-volume" pod "87dfb15b-fa89-461c-b36a-bceb692300c0" (UID: "87dfb15b-fa89-461c-b36a-bceb692300c0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:17.479469 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.479431 5074 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dfb15b-fa89-461c-b36a-bceb692300c0-kube-api-access-d2bcw" pod "87dfb15b-fa89-461c-b36a-bceb692300c0" (UID: "87dfb15b-fa89-461c-b36a-bceb692300c0"). InnerVolumeSpecName "kube-api-access-d2bcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:17.577885 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.577846 5074 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2bcw\" (UniqueName: \"kubernetes.io/projected/87dfb15b-fa89-461c-b36a-bceb692300c0-kube-api-access-d2bcw\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:17.577885 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:17.577880 5074 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/87dfb15b-fa89-461c-b36a-bceb692300c0-extensions-socket-volume\") on node \"ip-10-0-135-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:18.323720 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:18.323677 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" event={"ID":"87dfb15b-fa89-461c-b36a-bceb692300c0","Type":"ContainerDied","Data":"460b355d6592e5e3c2e7d8eee5e818bd08cc03bb4256ad709c5c51f7eb93bcb7"} Oct 31 14:03:18.323720 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:18.323715 5074 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j" Oct 31 14:03:18.324242 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:18.323779 5074 scope.go:122] "RemoveContainer" containerID="91369d75f398460a3e9e1b0e44645d6884f03af6bafec1b8d5672a80bb693649" Oct 31 14:03:18.343496 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:18.343466 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j"] Oct 31 14:03:18.348878 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:18.348854 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jsr6j"] Oct 31 14:03:19.937700 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:03:19.937654 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87dfb15b-fa89-461c-b36a-bceb692300c0" path="/var/lib/kubelet/pods/87dfb15b-fa89-461c-b36a-bceb692300c0/volumes" Oct 31 14:03:25.981338 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:03:25.981297 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:03:55.981732 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:03:55.981694 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:04:17.760659 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.760587 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd"] Oct 31 14:04:17.761225 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.761183 5074 cpu_manager.go:414] "containerMap: removing container" podUID="87dfb15b-fa89-461c-b36a-bceb692300c0" containerName="manager" Oct 31 14:04:17.761225 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.761206 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87dfb15b-fa89-461c-b36a-bceb692300c0" containerName="manager" Oct 31 14:04:17.761357 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.761235 5074 cpu_manager.go:414] "containerMap: removing container" podUID="29010444-d00c-4f5e-91d9-706a5f8900db" containerName="revoke-keys" Oct 31 14:04:17.761357 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.761244 5074 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29010444-d00c-4f5e-91d9-706a5f8900db" containerName="revoke-keys" Oct 31 14:04:17.761463 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.761381 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="29010444-d00c-4f5e-91d9-706a5f8900db" containerName="revoke-keys" Oct 31 14:04:17.761463 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.761407 5074 memory_manager.go:365] "RemoveStaleState removing state" podUID="87dfb15b-fa89-461c-b36a-bceb692300c0" containerName="manager" Oct 31 14:04:17.764569 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.764545 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:17.769021 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.768991 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-58c4z\"" Oct 31 14:04:17.786832 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.786795 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd"] Oct 31 14:04:17.865482 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.865435 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/628c72cd-fbea-45cc-8080-a1b482b787f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dj5qd\" (UID: \"628c72cd-fbea-45cc-8080-a1b482b787f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:17.865706 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.865561 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c9qn\" (UniqueName: \"kubernetes.io/projected/628c72cd-fbea-45cc-8080-a1b482b787f8-kube-api-access-5c9qn\") pod \"kuadrant-operator-controller-manager-678d87766f-dj5qd\" (UID: \"628c72cd-fbea-45cc-8080-a1b482b787f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:17.966797 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.966759 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c9qn\" (UniqueName: \"kubernetes.io/projected/628c72cd-fbea-45cc-8080-a1b482b787f8-kube-api-access-5c9qn\") pod \"kuadrant-operator-controller-manager-678d87766f-dj5qd\" (UID: \"628c72cd-fbea-45cc-8080-a1b482b787f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:17.966976 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.966919 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/628c72cd-fbea-45cc-8080-a1b482b787f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dj5qd\" (UID: \"628c72cd-fbea-45cc-8080-a1b482b787f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:17.967283 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.967264 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/628c72cd-fbea-45cc-8080-a1b482b787f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dj5qd\" (UID: \"628c72cd-fbea-45cc-8080-a1b482b787f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:17.979785 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:17.979751 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c9qn\" (UniqueName: \"kubernetes.io/projected/628c72cd-fbea-45cc-8080-a1b482b787f8-kube-api-access-5c9qn\") pod \"kuadrant-operator-controller-manager-678d87766f-dj5qd\" (UID: \"628c72cd-fbea-45cc-8080-a1b482b787f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:18.084405 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:18.084296 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:18.370912 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:18.370887 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd"] Oct 31 14:04:18.373347 ip-10-0-135-41 kubenswrapper[5074]: W1031 14:04:18.373316 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628c72cd_fbea_45cc_8080_a1b482b787f8.slice/crio-abea7fc142f4af3c57e383e06c8da5a2cebd548e2736b1a6db432b3eebef3747 WatchSource:0}: Error finding container abea7fc142f4af3c57e383e06c8da5a2cebd548e2736b1a6db432b3eebef3747: Status 404 returned error can't find the container with id abea7fc142f4af3c57e383e06c8da5a2cebd548e2736b1a6db432b3eebef3747 Oct 31 14:04:18.629041 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:18.628929 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" event={"ID":"628c72cd-fbea-45cc-8080-a1b482b787f8","Type":"ContainerStarted","Data":"9c4982a09f854efcbd315702330c95498aa7c8d7f26e18496d61b3879be67040"} Oct 31 14:04:18.629041 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:18.629030 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" event={"ID":"628c72cd-fbea-45cc-8080-a1b482b787f8","Type":"ContainerStarted","Data":"abea7fc142f4af3c57e383e06c8da5a2cebd548e2736b1a6db432b3eebef3747"} Oct 31 14:04:18.629257 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:18.629063 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:18.651113 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:18.651049 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" podStartSLOduration=1.651035876 podStartE2EDuration="1.651035876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:04:18.650474248 +0000 UTC m=+5552.828192232" watchObservedRunningTime="2026-10-31 14:04:18.651035876 +0000 UTC m=+5552.828753860" Oct 31 14:04:25.981229 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:04:25.981192 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:04:29.637499 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:29.637464 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dj5qd" Oct 31 14:04:41.136144 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:41.136095 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:04:41.764800 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:41.764764 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:04:41.765067 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:41.765042 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:04:54.173019 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:54.172978 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:04:55.982117 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:04:55.982076 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:04:58.759109 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:04:58.759060 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:05:01.646824 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:01.646784 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:05:19.708443 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.708405 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f787c88f-9tbzd"] Oct 31 14:05:19.712090 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.712070 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.719203 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.719175 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 14:05:19.719360 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.719239 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 14:05:19.719655 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.719599 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dbe407-bb02-4fec-bda5-fd53e632ddb6-maas-api-tls\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.719767 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.719700 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbe407-bb02-4fec-bda5-fd53e632ddb6-metrics-certs\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.719835 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.719793 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pbh\" (UniqueName: \"kubernetes.io/projected/86dbe407-bb02-4fec-bda5-fd53e632ddb6-kube-api-access-79pbh\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.719917 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.719896 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dbe407-bb02-4fec-bda5-fd53e632ddb6-openshift-service-ca\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.720845 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.720822 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhftr\"" Oct 31 14:05:19.723954 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.723928 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f787c88f-9tbzd"] Oct 31 14:05:19.806386 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.806350 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:05:19.806558 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.806449 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:05:19.821082 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.821048 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dbe407-bb02-4fec-bda5-fd53e632ddb6-maas-api-tls\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.821218 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.821105 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbe407-bb02-4fec-bda5-fd53e632ddb6-metrics-certs\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.821218 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.821163 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pbh\" (UniqueName: \"kubernetes.io/projected/86dbe407-bb02-4fec-bda5-fd53e632ddb6-kube-api-access-79pbh\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.821291 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.821225 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dbe407-bb02-4fec-bda5-fd53e632ddb6-openshift-service-ca\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.821989 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.821962 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dbe407-bb02-4fec-bda5-fd53e632ddb6-openshift-service-ca\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.823554 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.823528 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbe407-bb02-4fec-bda5-fd53e632ddb6-metrics-certs\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.823725 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.823705 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dbe407-bb02-4fec-bda5-fd53e632ddb6-maas-api-tls\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:19.834205 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:19.834181 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pbh\" (UniqueName: \"kubernetes.io/projected/86dbe407-bb02-4fec-bda5-fd53e632ddb6-kube-api-access-79pbh\") pod \"maas-api-f787c88f-9tbzd\" (UID: \"86dbe407-bb02-4fec-bda5-fd53e632ddb6\") " pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:20.026279 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.026227 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:20.248159 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.248118 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:05:20.316752 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.316533 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f787c88f-9tbzd"] Oct 31 14:05:20.320509 ip-10-0-135-41 kubenswrapper[5074]: W1031 14:05:20.320462 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dbe407_bb02_4fec_bda5_fd53e632ddb6.slice/crio-95161c0258a2858ad97b774674210f5b0062cdcb3661e93005b270ee0ee6ecc8 WatchSource:0}: Error finding container 95161c0258a2858ad97b774674210f5b0062cdcb3661e93005b270ee0ee6ecc8: Status 404 returned error can't find the container with id 95161c0258a2858ad97b774674210f5b0062cdcb3661e93005b270ee0ee6ecc8 Oct 31 14:05:20.322061 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.322039 5074 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:05:20.967388 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.967332 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" event={"ID":"86dbe407-bb02-4fec-bda5-fd53e632ddb6","Type":"ContainerStarted","Data":"5af97a49f7f23e7167df5deae5a590192c0bee29c2449dc8bc3b47e15ddf1cb9"} Oct 31 14:05:20.967858 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.967412 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" event={"ID":"86dbe407-bb02-4fec-bda5-fd53e632ddb6","Type":"ContainerStarted","Data":"95161c0258a2858ad97b774674210f5b0062cdcb3661e93005b270ee0ee6ecc8"} Oct 31 14:05:20.968352 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.968326 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:20.991532 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:20.991464 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" podStartSLOduration=1.638924402 podStartE2EDuration="1.991442265s" totalImagesPullingTime="352.517863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:20.988989059 +0000 UTC m=+5615.166707044" watchObservedRunningTime="2026-10-31 14:05:20.991442265 +0000 UTC m=+5615.169160250" Oct 31 14:05:26.048747 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:05:26.046594 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:05:26.768892 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:26.768834 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:26.769232 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:26.769180 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:28.008893 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:28.008860 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f787c88f-9tbzd" Oct 31 14:05:49.971304 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:49.971263 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:05:55.378241 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:05:55.378207 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:05:55.987797 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:05:55.987747 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:06:09.940949 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:09.940910 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:06:15.064866 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:15.064823 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:06:25.983570 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:06:25.983528 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:06:36.073416 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:36.073373 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:06:46.342558 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:46.342522 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:06:47.369411 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:47.369299 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 14:06:47.397184 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:47.397160 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 14:06:51.772416 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:51.772377 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:06:54.032656 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:54.031979 5074 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n"] Oct 31 14:06:54.039217 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:54.039182 5074 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771298d6-lvk5n"] Oct 31 14:06:55.939702 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:55.939663 5074 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b567bef2-e9db-42de-82a4-e2dea9f7a636" path="/var/lib/kubelet/pods/b567bef2-e9db-42de-82a4-e2dea9f7a636/volumes" Oct 31 14:06:55.983292 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:06:55.983248 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:06:56.771897 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:06:56.771859 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:07:12.872821 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:12.872783 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:07:23.135954 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:23.135911 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:07:25.996439 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:07:25.996381 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:07:27.410293 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:27.409750 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-vdjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:07:27.410293 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:27.409852 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-vdjwd" podUID="8b89ff01-1a45-4ec2-b5bd-75c7178126f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:07:27.668718 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:27.668596 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:07:41.138581 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:41.138535 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:07:55.982606 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:07:55.982568 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:07:57.365158 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:57.365118 5074 scope.go:122] "RemoveContainer" containerID="48fd60e43aa099b4cc84ee111bc7a797d8c788a1df22bf1dd0f9212d5cd34ee3" Oct 31 14:07:58.134976 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:07:58.134934 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:08:25.983201 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:08:25.983161 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:08:55.982194 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:08:55.982156 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:08:57.385989 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:08:57.385948 5074 scope.go:122] "RemoveContainer" containerID="73a5d135fb1a84fb803eee30c07ca5979da8ceae04df1b9d80cddd316eb158bd" Oct 31 14:09:11.758797 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:09:11.758754 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:11.759214 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:09:11.758855 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:25.981771 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:09:25.981726 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:09:41.758803 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:09:41.758754 5074 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-5wltg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:41.759498 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:09:41.758821 5074 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-5wltg" podUID="19865d77-5b0f-456a-a09e-b1922459c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:50.178756 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:09:50.178717 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6jt82"] Oct 31 14:09:55.981769 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:09:55.981732 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:10:11.964651 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:11.964597 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tpdfx_c39defbd-ec41-4ab3-a577-ba969686c319/manager/0.log" Oct 31 14:10:12.163966 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:12.163927 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xtpgb_a830a12a-e88b-4cca-97f6-8aa69759dfbd/manager/0.log" Oct 31 14:10:14.150688 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:14.150652 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f787c88f-9tbzd_86dbe407-bb02-4fec-bda5-fd53e632ddb6/maas-api/0.log" Oct 31 14:10:19.787081 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:19.787044 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r6gnb_6673fd3f-afa0-4106-901a-7afcad6cdeee/manager/0.log" Oct 31 14:10:20.387546 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:20.387498 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dj5qd_628c72cd-fbea-45cc-8080-a1b482b787f8/manager/0.log" Oct 31 14:10:20.627015 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:20.626983 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6jt82_fbeeade1-58fc-47fb-b398-6547ad67129b/limitador/0.log" Oct 31 14:10:21.766062 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:21.766025 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-657b74449cczhbq_7284a125-b30e-41ef-b402-9d43ae1e75ad/istio-proxy/0.log" Oct 31 14:10:22.487753 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:22.487708 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-5wltg_19865d77-5b0f-456a-a09e-b1922459c50a/istio-proxy/0.log" Oct 31 14:10:23.844774 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:23.844717 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-szf5x_89558c79-c272-4647-a36f-8130cb513976/istio-proxy/0.log" Oct 31 14:10:24.046030 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:24.045990 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-vdjwd_8b89ff01-1a45-4ec2-b5bd-75c7178126f2/istio-proxy/0.log" Oct 31 14:10:24.454505 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:24.454475 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk_20266055-9cae-476d-8268-8f20300ff729/istio-proxy/0.log" Oct 31 14:10:25.007900 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:25.007860 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-865bf4d8bc-d5gwq_579b4046-76c9-4d7e-b77a-9888510a22a8/router/0.log" Oct 31 14:10:25.981956 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:10:25.981919 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:10:42.182069 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:42.182015 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sk7rm_8cea9a4e-e914-40b6-9237-345be4aff28c/global-pull-secret-syncer/0.log" Oct 31 14:10:42.288549 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:42.288515 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6ps96_5b057b6d-b9d1-4642-95f3-41ba5498c24d/connection-checker/0.log" Oct 31 14:10:42.315294 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:42.315264 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hxg8d_c765c0f2-1bc6-4eee-9fde-317749984f5c/connection-checker/0.log" Oct 31 14:10:42.341325 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:42.341298 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tn6pm_e8d6247b-578f-4ee3-8fb4-01bb7da39a7d/connection-checker/0.log" Oct 31 14:10:42.513593 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:42.513559 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b4cc4_0554ee01-1433-4bd7-9a51-d74c757af46a/konnectivity-agent/0.log" Oct 31 14:10:42.637076 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:42.637036 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-41.us-east-2.compute.internal_f84469bc02ae1e7d219895e3165c40bd/haproxy/0.log" Oct 31 14:10:49.809245 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:49.809208 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r6gnb_6673fd3f-afa0-4106-901a-7afcad6cdeee/manager/0.log" Oct 31 14:10:50.007933 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:50.007899 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dj5qd_628c72cd-fbea-45cc-8080-a1b482b787f8/manager/0.log" Oct 31 14:10:50.038430 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:50.038381 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6jt82_fbeeade1-58fc-47fb-b398-6547ad67129b/limitador/0.log" Oct 31 14:10:52.055184 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:52.055149 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-vzqxc_dee0dda1-07c9-43fc-b7d6-f09b502d0149/cluster-monitoring-operator/0.log" Oct 31 14:10:52.335535 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:52.335454 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgf2c_07cde39d-85c9-4b82-9248-c57a806fcbe2/node-exporter/0.log" Oct 31 14:10:52.358323 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:52.358286 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgf2c_07cde39d-85c9-4b82-9248-c57a806fcbe2/kube-rbac-proxy/0.log" Oct 31 14:10:52.387625 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:52.387599 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgf2c_07cde39d-85c9-4b82-9248-c57a806fcbe2/init-textfile/0.log" Oct 31 14:10:52.904945 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:52.904914 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7dq5p_486b788d-7531-41c1-89d5-996d6a05c0d9/prometheus-operator-admission-webhook/0.log" Oct 31 14:10:53.620054 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.620013 5074 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449"] Oct 31 14:10:53.623963 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.623941 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.632313 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.632288 5074 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jr74b\"/\"default-dockercfg-c88j5\"" Oct 31 14:10:53.633837 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.633813 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449"] Oct 31 14:10:53.703480 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.703447 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr74b\"/\"kube-root-ca.crt\"" Oct 31 14:10:53.703663 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.703486 5074 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr74b\"/\"openshift-service-ca.crt\"" Oct 31 14:10:53.761488 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.761444 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-lib-modules\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.761675 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.761505 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-proc\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.761675 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.761651 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wbh\" (UniqueName: \"kubernetes.io/projected/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-kube-api-access-89wbh\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.761808 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.761679 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-sys\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.761808 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.761708 5074 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-podres\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863261 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863225 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wbh\" (UniqueName: \"kubernetes.io/projected/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-kube-api-access-89wbh\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863261 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863265 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-sys\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863519 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863290 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-podres\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863519 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863337 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-lib-modules\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863519 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863356 5074 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-proc\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863519 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863382 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-sys\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863519 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863440 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-proc\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863519 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863447 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-podres\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.863519 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.863489 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-lib-modules\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.875486 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.875419 5074 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wbh\" (UniqueName: \"kubernetes.io/projected/ddddbde7-e03e-48f0-bd56-ba1c9a69bc76-kube-api-access-89wbh\") pod \"perf-node-gather-daemonset-vz449\" (UID: \"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76\") " pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:53.937580 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:53.937551 5074 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:54.212243 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:54.212207 5074 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449"] Oct 31 14:10:54.215099 ip-10-0-135-41 kubenswrapper[5074]: W1031 14:10:54.215067 5074 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podddddbde7_e03e_48f0_bd56_ba1c9a69bc76.slice/crio-a394e33ccb6168c180321d647040c0b3e0278c336419e7b514bbbc8e14bc34fc WatchSource:0}: Error finding container a394e33ccb6168c180321d647040c0b3e0278c336419e7b514bbbc8e14bc34fc: Status 404 returned error can't find the container with id a394e33ccb6168c180321d647040c0b3e0278c336419e7b514bbbc8e14bc34fc Oct 31 14:10:54.457379 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:54.457335 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tpdfx_c39defbd-ec41-4ab3-a577-ba969686c319/manager/0.log" Oct 31 14:10:54.894346 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:54.894309 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-pnzjp_698d7a34-d950-442b-b12e-b0954c8fe873/networking-console-plugin/0.log" Oct 31 14:10:55.042079 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:55.042032 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" event={"ID":"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76","Type":"ContainerStarted","Data":"bb9ed8dc954a8470541f38d7f17611f566f9e55cd302f6db6198f973dd4eb5e1"} Oct 31 14:10:55.042295 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:55.042105 5074 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" event={"ID":"ddddbde7-e03e-48f0-bd56-ba1c9a69bc76","Type":"ContainerStarted","Data":"a394e33ccb6168c180321d647040c0b3e0278c336419e7b514bbbc8e14bc34fc"} Oct 31 14:10:55.042295 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:55.042240 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:10:55.066232 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:55.066179 5074 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" podStartSLOduration=2.06614784 podStartE2EDuration="2.06614784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:55.064031236 +0000 UTC m=+5949.241749222" watchObservedRunningTime="2026-10-31 14:10:55.06614784 +0000 UTC m=+5949.243865825" Oct 31 14:10:55.636853 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:55.636767 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/2.log" Oct 31 14:10:55.648286 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:55.648258 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vnxg_a8053559-57b0-468c-a369-893764b0ed4d/console-operator/3.log" Oct 31 14:10:55.879873 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:55.879838 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xtpgb_a830a12a-e88b-4cca-97f6-8aa69759dfbd/manager/0.log" Oct 31 14:10:55.983146 ip-10-0-135-41 kubenswrapper[5074]: E1031 14:10:55.983107 5074 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc7gp" containerName="extractor" Oct 31 14:10:56.429356 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:56.429274 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-fmzxl_9551cad5-7a59-43fe-af7a-ff0ed70a8789/download-server/0.log" Oct 31 14:10:57.430256 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:57.430215 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-sxrj4_719b7f16-3b6a-402e-8301-7b498a9ad131/volume-data-source-validator/0.log" Oct 31 14:10:59.171445 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:59.171414 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4psl_8c751f98-9133-4738-9052-8dba6020cda6/dns/0.log" Oct 31 14:10:59.193663 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:59.193619 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4psl_8c751f98-9133-4738-9052-8dba6020cda6/kube-rbac-proxy/0.log" Oct 31 14:10:59.279125 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:59.279098 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9v4bh_a069e898-8264-4573-b885-2528dadb0305/dns-node-resolver/0.log" Oct 31 14:10:59.686888 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:10:59.686847 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tpdfx_c39defbd-ec41-4ab3-a577-ba969686c319/manager/0.log" Oct 31 14:11:00.021325 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:00.021277 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cb45b6b6f-8pjrj_948aab57-f974-48a1-b43a-cb75577e8d7a/registry/0.log" Oct 31 14:11:00.088062 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:00.088027 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qkx64_0bcc043c-d8ac-4f2a-ae8d-3e233db2842d/node-ca/0.log" Oct 31 14:11:01.063097 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:01.063062 5074 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jr74b/perf-node-gather-daemonset-vz449" Oct 31 14:11:01.111990 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:01.111938 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xtpgb_a830a12a-e88b-4cca-97f6-8aa69759dfbd/manager/0.log" Oct 31 14:11:01.262024 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:01.261971 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-657b74449cczhbq_7284a125-b30e-41ef-b402-9d43ae1e75ad/istio-proxy/0.log" Oct 31 14:11:03.388876 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:03.388835 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-5wltg_19865d77-5b0f-456a-a09e-b1922459c50a/istio-proxy/0.log" Oct 31 14:11:03.731987 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:03.731923 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-szf5x_89558c79-c272-4647-a36f-8130cb513976/istio-proxy/0.log" Oct 31 14:11:03.764692 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:03.764660 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-vdjwd_8b89ff01-1a45-4ec2-b5bd-75c7178126f2/istio-proxy/0.log" Oct 31 14:11:03.846249 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:03.846192 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk_20266055-9cae-476d-8268-8f20300ff729/istio-proxy/0.log" Oct 31 14:11:04.002513 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:04.002434 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-865bf4d8bc-d5gwq_579b4046-76c9-4d7e-b77a-9888510a22a8/router/0.log" Oct 31 14:11:05.397293 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:05.397260 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tpdfx_c39defbd-ec41-4ab3-a577-ba969686c319/manager/0.log" Oct 31 14:11:06.790871 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:06.790840 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xtpgb_a830a12a-e88b-4cca-97f6-8aa69759dfbd/manager/0.log" Oct 31 14:11:06.836342 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:06.836308 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-657b74449cczhbq_7284a125-b30e-41ef-b402-9d43ae1e75ad/istio-proxy/0.log" Oct 31 14:11:07.513077 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:07.513043 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqcvc_e040a356-8196-40c6-8f2c-a94bb4c6bfe4/serve-healthcheck-canary/0.log" Oct 31 14:11:08.352310 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.352269 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-tc2tp_55bcf0cc-7eef-4cbe-9d72-140932580c56/insights-operator/1.log" Oct 31 14:11:08.356732 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.356700 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-tc2tp_55bcf0cc-7eef-4cbe-9d72-140932580c56/insights-operator/2.log" Oct 31 14:11:08.566468 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.566436 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-5wltg_19865d77-5b0f-456a-a09e-b1922459c50a/istio-proxy/0.log" Oct 31 14:11:08.567660 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.567617 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jc7gp_e60a880f-7773-41dd-9995-3b1719fe8878/kube-rbac-proxy/0.log" Oct 31 14:11:08.598553 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.598528 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jc7gp_e60a880f-7773-41dd-9995-3b1719fe8878/exporter/0.log" Oct 31 14:11:08.624295 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.624230 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jc7gp_e60a880f-7773-41dd-9995-3b1719fe8878/extractor/0.log" Oct 31 14:11:08.671074 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.671022 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-t6v9c-694pl_8e27f88c-9de4-4ad9-93d0-bca2279b39ab/insights-gathering/0.log" Oct 31 14:11:08.712842 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.712812 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-vsfcv-rjmh5_1e503e6f-5c7e-432f-b495-090af8eaf6dd/insights-gathering/0.log" Oct 31 14:11:08.739091 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.739043 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-szf5x_89558c79-c272-4647-a36f-8130cb513976/istio-proxy/0.log" Oct 31 14:11:08.761462 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.761436 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-vdjwd_8b89ff01-1a45-4ec2-b5bd-75c7178126f2/istio-proxy/0.log" Oct 31 14:11:08.807743 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.807721 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-648fc44f87-q6tjk_20266055-9cae-476d-8268-8f20300ff729/istio-proxy/0.log" Oct 31 14:11:08.887901 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:08.887834 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-865bf4d8bc-d5gwq_579b4046-76c9-4d7e-b77a-9888510a22a8/router/0.log" Oct 31 14:11:10.867529 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:10.867498 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-657b74449cczhbq_7284a125-b30e-41ef-b402-9d43ae1e75ad/istio-proxy/0.log" Oct 31 14:11:11.882573 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:11.882539 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c44b469d4-lm5gk_3716314e-4607-45e8-898d-fe7505c8f7fe/manager/0.log" Oct 31 14:11:11.908486 ip-10-0-135-41 kubenswrapper[5074]: I1031 14:11:11.908458 5074 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c44b469d4-lm5gk_3716314e-4607-45e8-898d-fe7505c8f7fe/copy-manifests/0.log"